The Good and Bad of Multiple-Choice Testing

The Testing Effect refers to the improvement of students' performance when students are tested repeatedly of their knowledge. See previous blog entry. Frequent testing using multiple choice questions has also been shown to be effective not just for recall but for higher Bloom level of learning. So the testing effect is not limited to memory recall of facts only, but also to application type of learning. However, the presence of incorrect answers (also known as "lures") in multiple choice questions may cause the students to acquire incorrect concepts via faulty reasoning. Even then, repeated testings produce more positive benefits than negative side-effects.

One way to compensate the negatives of multiple-choice testing is to provide immediate feedback to correct learner's misconceptions and avoid their construction of incorrect knowledge. Another way is to offer a "don't know" option or a penalty for selecting a wrong answer. This can also reduce the amount of guessing. Lastly, a different way of testing may be used, such as short answer questions, which seem to have even more positive benefits than multiple choice questions.

Reference:

Marsh, E., Roediger III, H., Bjork, R., Bjork, E. (2007). The Memorial Consequences of Multiple-Choice Testing. Psychonomic Bulletin & Review. 14(2), 194-199.

Advice to Students: do more testing and less studying!!

At least for memory recall, taking a memory test repeatedly rather than studying repeatedly results in much better long term retention. The abstract from the first reference below says it all:

Taking a memory test not only assesses what one knows, but also enhances later retention, a phenomenon known as the testing effect. We studied this effect with educationally relevant materials and investigated whether testing facilitates learning only because tests offer an opportunity to restudy material. In two experiments, students studied prose passages and took one or three immediate free-recall tests, without feedback, or restudied the material the same number of times as the students who received tests. Students then took a final retention test 5 min, 2 days, or 1 week later. When the final test was given after 5 min, repeated studying improved recall relative to repeated testing. However, on the delayed tests, prior testing produced substantially greater retention than studying, even though repeated studying increased students’ confidence in their ability to remember the material.Testing is a powerful means of improving learning, not just assessing it.

In other words, if S stands for study, and T stands for testing, a final recall test after the sequence STTT results in much higher retention than SSST or SSSS. In computer science, most of the learning requires reasoning rather than memory recall, although a good repository of learned concepts is definitely an asset to being a good programmer. However, from a number of interviews with students enrolled in a first year programming course, when asked how they prepared for exams, 90% of the students would say reading from lecture notes, textbooks, and only about 10% would mention about doing some coding and testing. The learning-by-experimentation concept seems to be foreign to many students.

It won't be surprising that the result from Roediger and Karpicke applies just as well to reasoning skills as memory recall. What will be interesting for CS is to identify the set of core skills and concepts that expert programmers need to have and apply this strategy of studying and testing (mostly) throughout a program of study rather than just a course, and conduct longitudinal study of their retention and programming skills beyond graduation. Also, how can repeat testing be made "fun" for learners? Is there an "optimal" study and test sequence for CS courses?

References:

Roediger III, H., Karpicke, J. (2006). Test-Enhanced Learning. Psychological Science. 17(3), pp 249 - 255.

Karpicke, J., Roediger III, H. (2008). The Critical Importance of Retrieval for Learning. Science. Vol 319, pp 966 - 968. Link.

Two-stage Cooperative Exams

The idea of a two-staged cooperative exam is that students take the same exam repeatedly during an extended period of time but in different settings. These settings can be individual in the beginning, then working in pairs, or collaboratively in a larger group. The goal is to turn these testing sessions into a learning experience.

Here is an example of how this is implemented in a large class for midterm or final exams: during the first 30 minutes of the class period, the students take a multiple-choice exam with about 20 - 25 questions in it individually. They hand in the answer sheets at the end of the exam. Then right away, they are given the same multiple-choice exam but with added questions in it, and are asked to work on it collaboratively with someone close by for 45 minutes. They can use books, notes, and other resources. The grade of the exam is calculated based on a weighted average (75%) of the first submission and 25% of the second submission of the exam. However, if this grade is less than the grade in the first submission (i.e. from the solo effort alone), then the final score of this exam is based solely on the first submission.

With this simple change in exam format throughout the term, there has been large improvement in the final exam scores from a mean of 74% to 80%, based only on the solo part of the exam. Although it seems that the collaborative component of the exam may have boosted the final score, a statistical comparison with grades from previous years with no collaborative component in the exams shows that there is no dramatic change in grade distribution. The number of students at the bottom rungs of the ladder are fewer with the two-stage cooperative exam strategy, but there is no increase in the upper rungs.

References:

Yuretich, R., Khan, S., Leckie, R., Clement. (March 2001). Active-Learning Methods to Improve Student Performance and Scientific Interest in a Large Introductory Oceanography Course. Journal of Geoscience Education. 49(2), p 111- 119.

Yuretich, R. Accessing Higher-Order Thinking in Large Introductory Science Classes.

"Well Polished" Lectures May Not be Good for Learners

Most trainers are "conditioned" to expect improvements in the performance and also the "happiness" of their trainees.  Instructors don't see how their students perform after a course, but during the course, it is important for them, as well as for the students, to see that progress is being made with measurable improvements.  However, most training methods produce impressive short term improvements with no long term benefits.  As an example, practice drilling exercises may give the impression that the students have actually acquired a set of knowledge and skills through increasing familiarity of the material.  However, it has been shown that the more familiar the learners believe they are with a subject, their actual level of comprehension is actually inversely related (as least in certain domains like physics or music).  As another example, if answers are given readily, learners adopt a mentality that they "knew it all along".  Hence a well polished lecture where the listeners can follow easily may give the illusion that the learners have already learned or known the material, which in fact, may not be the case when they are called upon the task to actually solving a problem based on the material presented.

Bjork gives five examples of training that may produce durable transfer of knowledge and skills in post-training environments. This implies long term retention and transfer of knowledge and skills to new situations.  The key is to introduce meaningful and desirable difficulties in the training.  Here are the five examples:

• Varying the conditions of practice such as the (un)predictability of the training environment, scheduling practice exercises in variety and in random fashion rather than a block of training on one specific task, etc.
• Providing contextual interference such as designing and interleaving materials to be learned, rearranging the material presentation that is inconsistent with an outline, or adding to the complexity of the tasks to be performed.
• Distributing practice on a given task over time rather than "cramming" the material in a short session.
• Reducing feedback to the learner (mainly applicable to motorized skill).
• Using tests as learning events rather than providing more study opportunities.

In Computer Science, the lab exercises and problem sets do help students in their learning, especially when the complexity and context of the problems are varied.  By mixing the type of problems to be solved, changing the duration between similar types problems to be solved, and the use of tests to continue monitor student progress, not just within a course, but over several courses, we may begin to get a clearer picture of our student learning.

Reference:

Bjork, R. (1994). Memory and metamemory considerations in the training of human beings.  In J. Metcalfe and A. Shimamura (Eds.), Metacognition: Knowing About Knowing (pp. 185-205). Cambridge, MA: MIT Press.

Demos .. learning or entertainment?

Doing live code demos in class can help in engaging students in their learning. Students get to see the effects of code changes right away rather than just hearing about the concepts through a lecture presentation. However, at least for physics, students who observe demonstrations perform only slightly better, but not statistically significant, than those who do not. Without active participation in the demonstration, students are not engaging with their learning, just like listening passively in a lecture. The only difference is that the demonstration may be a bit more entertaining and may have some affective effects on the students.

To make the most out of classroom demonstrations, one simple strategy is to simply ask the students to predict what will happen before doing the demonstration. Adding one or two minutes in having the students think about the topic under discussion, and predict what would happen if there is a change, turns out to have dramatic effects on student learning. If students are also required to discuss their predictions with their peers, their learning can be improved even more.

Reference:

Crouch, C., Fagen, A., Callan, J., Mazur, E. (June 2004). Classroom demonstration: Learning tools or entertainment?. American Association of Physics Teachers. 72(6), p 835 - 838.

What students need to know to solve Math / CS problems?

According to Richard Mayer, there are four essential stages one needs to go through to solve a typical mathematical problem like the following:

Floor tiles are sold in squares 30 centimeters on each side. How much world it cost to tile a rectangular room 7.2 meters long and 5.4 meters wide if the tiles cost $.72 each?

The four stages are: problem translation which converts each sentence into an internal representation, problem integration which involves putting the different pieces of information into a coherent whole, solution planning which involves the selection of the most appropriate strategy to solve the problem, and solution execution which involves carrying out the procedures to derive the solution.

The same four stages of problem solving can be applied to solving Computer Science problems. Problem translation involves a good data representation, usually in the form of a data structure, database design, or file structure. Once we have a good data representation, the flow of data and its transformation need to be analyzed. A good strategy usually in the form of a good algorithm needs to be selected next, and finally, the problem can be coded in a programming language for execution.

In CS, we have at least a course for each of these steps: Data Structure, Systems Analysis and Design / Software Engineering, Algorithm Design, and Programming Languages. The process of solving problems in computer science is indeed non-trivial, but yet, in many CS1 courses, we expect our students to be able to do all these while the primary focus of the course is probably just getting students to learn a programming language. Since the first course in CS usually determines whether a student will continue into the discipline, care must be taken not to overwhelm the students with too much material and frustrate them in their learning.

Reference:

Mayer, R. (2007) Learning and Instruction. Prentice Hall.

Learning Focused Course Transformation

At the United States Air Force Academy, a learning-focused transformation of Biology and Physics core courses was made to support deep student learning. This involves first transforming the learning goals from using terms like "list, find, calculate, describe, use, what, and when" to "explain, analyze, apply, create, predict, and evaluate". Students are also exposed to familiar and concrete settings where the knowledge can be applied. E.g. students are asked to serve as "expert witness" in a trial which requires their knowledge on gene expression, or they have to explain how spies can tap phone lines during the Cold War using Faraday's law. Class lessons include mini-lectures (about 10 minutes) and the rest of the time is mostly spent on learning experiences through activities and exploration. Instructors become learning facilitators rather than just lecturers.

The ultimate question is whether students learn just as much from activity based lessons as from traditional lecture style of delivery of content. In Computer Science education, students are invariably exposed to learning through activity based programming assignments since the nature of the discipline is mostly practical and applied. However, programming is not the only activities that students can be involved in, even though it is the most natural one. Especially if one of the learning goals is the development of abstract thinking, care must be taken not to over-emphasize the programming aspect that may monopolize the time and attention the students should spend. Programming projects can take up a lot of time and the students may end up gaining programming skills and not other skills. Here is where concise learning goals need to be articulated and the proportion of time for each learning goal is matched appropriately with the learning activities.

Reference:

Sagendorf, K., Noyd, R., Morris, D. (2009). The Learning-Focused Transformation of Biology and Physics Core Courses at the U.S. Air Force Academy. Journal of College Science Teaching. January / February 2009.