Worked Examples

An important discovery of Cognitive Load Theory (CLT) (Sweller, 1988) is that studying partially worked examples provide better learning results for novices in computing than working through problems from scratch or studying completely worked examples. Gray et al. (2007) suggested the use of fading worked example as an effective strategy for lowering cognitive load in the novice phase of skill acquisition in programming education.

The idea of a fading worked example (FWE) is a sequence of partially worked examples in which each problem in the sequence contains one fewer worked step than its predecessor so that, in the end, the learner is given a problem to solve with no worked steps provided. Thus in systems programming, instructors may start with a fully worked example (Clark et al., 2006) from a problem statement, to analysis, design, coding and testing. Then the next example may involve all steps except coding. The next example may remove design, etc, until the students are required to solve a problem given just a problem statement.

The key to creating FWE is decomposition of each learning goal into smaller steps. As an example of using FWE for learning programming, each aspect of a programming language is identified. This includes variable, expression, assignment, iteration, subroutine call, etc. Next the use of each of these aspects in a program is related to the dimensions of problem solving, namely design, implementation and semantics.

How does studying worked examples compared to actual practice? Actively solving practice problems imposes much more mental work than reviewing worked examples. However, skipping study of worked examples may impose too much cognitive load on the learners when they try to jump into practice assignments right away. (See Guzdial blog entry.) Studies have shown that students who learned by doing took twice as much time to learn as students who learned from worked examples (Mayer, 2008, chapter 9). Students also benefit more with worked examples if they generate explanations as they study the worked examples (meta-cognitive skill development).

A compromise between worked examples and actual practice is a completion example where some of the steps are demonstrated in a worked example and the other steps are completed by the learner as in a practice problem.

It should be noted that as learners gain expertise, worked examples actually become detrimental and they are better off working all the problems. The worked examples can become redundant. This is where FWE will be most useful.

Reference:

Clark, R.C., Nguyen, and F., Sweller, J. (2006). Efficiency in Learning. San Francisco: Pfeiffer. (Chapter 8).

Gray, S., Clair, C., James, R., Mead, J. (2007). Suggestions for Graduated Exposure to Programming Concepts Using Fading Worked Examples. International Computing Education Research Workshop, Proceedings of the third international workshop on Computing education research. pp 99-110.

Mayer, R. E. (2008). Learning and Instruction (2nd ed). Upper Saddle River, NJ: Merrill Prentice-Hall.

Sweller, J. (1988). Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science. 12(2).

Video Lectures

Internet delivered video lectures have been found to prepare students for exams as effectively as live in-class lectures in a biology course (Lents and Cifuentes, 2009) although students were not enthused with the concept of video lectures initially. Another experiment with video podcasts for Java CS1 course resulted in less than expected participation (Murphy and Wolff, 2009). However, in yet another study, students in a first semester calculus-based mechanics course using multimedia modules not only learned more than students using traditional textbook presentation, but also retained information better (Stelzer et al. 2009).

Much effort has gone into research on the design of multimedia materials to improve learning. This includes designing materials to help students stay focused of the learning goals, use of different input channels (visual and auditory) to help students build meaning and understanding, offloading (presenting words as narration rather than on-screen text), weeding (eliminating interesting but extraneous material), signaling (adding arrows or highlighting for emphasis), and aligning words and pictures (Mayer, 2001) (Mayer, 2003).

References:

Lents, N., and Cifuentes, O. (November / December 2009). Web-Based Learning Enhancements: Video Lectures Through Voice-Over PowerPoint in a Majors-Level Biology Course. Journal of College Science Teaching. 39(2), pp 38 - 46.

Mayer, R.E. (2001). The Cambridge Handbook of Multimedia Learning. Cambridge U.P., Cambrdige.

Mayer, R.E. and Moreno, R. (2003). Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educational Psychologist. 38(1), pp 43 - 52.

Murphy, L. and Wolff, D. (2009). Creating Video Podcasts for CS1: Lessons Learned. NorthWest Academic Computing Consortium (NWACC), Journal of Computing Sciences in Colleges, 25(1). pp 152 - 158.

Stelzer, T., Gladding, G., Mestre, J., Brookes, D. (February 2009). Comparing the Efficacy of Multimedia Modules with Traditional Textbooks for Learning Introductory Physics Content. American Association of Physics Teachers. 77(2), pp 184 - 190.

Good Problems and Effective Structures for Groups

Context-rich group problems help students to focus on the concepts and principles that are needed to solve them. They have the following general characteristics:

• Problem statement does not always specify the unknown to be computed.
• More information may be available than is needed to solve the problem.
• Some of the information needed to solve the problem may be missing from the question. Students need to determine what the missing information is and how to come up with it.
• Reasonable assumptions may need to be made to simplify the problem and allow for a meaningful solution.

Groups of three and four members are found to generate better plans for solving problems and a solution with fewer conceptual mistakes than pairs. Pairs usually have no mechanism for deciding between two strongly held viewpoints. In groups of four, one student was invariably left out of the problem-solving process. That person is usually the most timid or the most knowledgeable.

Homogeneous gender groups and mixed gender groups of two females and one male performed better than groups with two males and one female.

Groups with mixed ability performed as well as groups consisting of only high-ability students (who tend to make problems more complicated than necessary or overlook the obvious), and better than groups with students of only low or medium ability. Low ability students contribute by keeping the groups on track by pointing out the obvious and simple ideas, and requesting for clarification of the concepts and procedures that are needed to solve the problems (which the higher ability students sometimes realize their wrong assumptions and mistakes when they justify their solutions to them).

To avoid dominance of any student in a group, or to avoid a group from jumping at the first possible solution to avoid conflict in the group, two strategies can be used:

1. have students take on special roles. In a three member group, the roles of Manager (who designs plans for action and suggests solutions), Skeptic (who questions premises and plans), and Checker / Recorder (who organizes and keeps track of the discussions) can be assigned.
2. have the students reflect on how well their groups have worked and suggest ways of improvement at the end of each activity.
   

Reference:

Heller, P., Hollabaugh, M. (July 1992). Teaching Problem Solving Through Cooperative Group. Part 2. Designing Problems and Structuring Groups. American Association of Physics Teachers. 60(7). pp 637 - 644.

Is Collaborative Group Learning Useful?

A study on the effectiveness of group problem solving was conducted by Heller et al. (1992). The instructional approach was as follows:

• students were taught general problem-solving strategies
• a set of context-rich practice and test problems were given to help students focus their attention on the need to use conceptual knowledge to analyze a problem
• students worked in carefully managed groups to practice solving context-rich problems

Students' work were judged based on "expert" level of problem solving which is characterized by the following:

• evidence of conceptual understanding
• usefulness of information identified to solve the problems
• match of equations with information identified
• reasonable plan
• logical progression
• appropriate mathematics

Results: Group problem solutions were significantly better than those produced by the best problem solvers from each group on matched problems. Individual problem solving performance also improved over time. The key seems to be explicit problem solving strategy instruction and having the students practice using the strategy in groups.

Reference:

Heller, P., Keith, R., Anderson, S. (July 1992). Teaching Problem Solving Through Cooperative Grouping. Part 1: Group versus Individual Problem Solving. American Association of Physics Teachers. 60(7). pp 627 - 636.

Student Cheating in CMS

Do students tend to cheat more when they write exams or quizzes using online course management systems (CMS) like WebCT or Blackboard? Not according to Charlesworth et al. (2006). They did a survey on 178 students and asked them first their definitions of cheating, and their main reasons to cheat in a typical classroom. Most students define cheating as copying or taking answers from others, and their major reasons for cheating include, in order of importance: 1) laziness, 2) grades, 3) pressure to do well and not fail, 4) lack of knowledge, and lastly, 5) opportunity. Given that students are not the best in making proper assessment of themselves, I am not sure if this list accurately ordered. Here is how I would re-interpret this list. As noted in the paper, "[m]any students report lengthy study sessions yet realize incomplete understanding due to factors such as poor study skills and lack of knowledge. As a result, students may feel unprepared for quizzes and examinations, and seek alternative methods to ensure success." If students do not grasp the material (i.e. lack of knowledge (4)), they feel pressured to succeed (3) to obtain good grades (2), but since hard work is difficult, some may give up, and blame it on their laziness (1), and given the right opportunity (5) to cheat, they would do so.

It is also interesting to note from the paper that students whose GPA is between 2.4 - 3.0 are more likely to cheat on written assignments. However, the study does not show that a web-enhanced course automatically increase the amount of cheating.

Reference:

Charlesworth, P., Charlesworth, D., Vician, C. (September 2006). Students' Perspectives of the Influence of Web-Enhanced Coursework on Incidences of Cheating. Journal of Chemical Education. 83(9), pp 1368 - 1375.

Problem Based Learning

Problem based learning (PBL) is not simply throwing a problem to the students and let them figure out the solutions all by themselves. There are significant support elements to guide the students in the learning. It is actually a well defined, structured instructional method that students work through in seven steps with appropriate scaffolding support, learning resources, instructional support, tutor support, group discussions, etc.:

1. students clarify any terms and concepts in the problem text
2. generate a definition of the problem (or what is really the problem to be solved)
3. students brainstorm ideas, hypothesize, question about the problem
4. systematize and scrutinize the ideas
5. produce a list of issues for individual learning (the learning goals / contents behind the problem)
   
6. the learning issues are used to guide student study activities where students study the available resources
7. students share findings, review and discuss literature, solve other problems, and synthesize what is learned.

PBL is therefore not equated to minimally guided instruction (Schmidt et al. 2007) (Hmelo-Silver et al, 2007).

References:

Hmelo-Silver, C., Duncan, R.G., Chinn, C.A. (2007). Scaffolding and Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006). Educational Psychologist. 42(2), pp 99 - 107.

Schmidt, H.G., Loyens, S.M., van Gog, T., Paas, F. (2007). Problem-Based Learning is Compatible with Human Cognitive Architecture: Commentary on Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), pp 91 - 97.

Minimal Guided Learning

Are there really any benefits to minimal guided learning, as practiced in a number of classroom activities in the form of inquiry learning, problem based learning, invention activities, etc.? According to Kirschner et al. (2006), not much. Their argument is that problem solving takes place in the working memory, which is severely limited in capacity when dealing with novel information, and since learning is to ultimately alter long term memory, they conclude that 1) the changes in the short term memory will likely not cause any changes in the long term memory since all information is lost within 30 seconds if the information is not rehearsed, 2) the heavy cognitive load is detrimental to learning.

Instead, a worked example with strongly guided instruction, process worksheets where descriptions on how to solve problems with specific hints and rules of thumb are more effective for student learning. Kyllonen and Lajoie (2003) found that highly structured instructional presentations benefit less able learners and unstructured instructional presentations benefit more able learners. Clark (1982) also noted that less able learners tend to choose less guided approaches to learning and they learn less. Higher aptitude students tend to choose more guided approaches to learning but they could have learned even more if they have chosen less guided instruction.

Is it possible then that CS education tends to create such a heavy cognitive load on our students, especially first year students, that result in such high attrition rate? Would providing students with detailed worked programming examples, strategies to solve programming problems, use of worksheets to allow students engage in deliberate practice help transition students to become more skilled programmers a better approach?

References:

Clark, R.E. (1982). Antagonism between Achievement and Enjoyment in ATI Studies. Educational Psychologist, 17, pp 92 - 101.

Kyllonen, P.C., and Lajoie, S.P. (2003). Reassessing aptitude: Introduction to a Special Issue in honor of Richard E. Snow. Educational Psychologist, 38, pp 79 - 83.

Kirschner, P.A., Sweller, J., Clark, R.E. (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching. Educational Psychologist, 41(2), pp 75 - 86.

Sweller, J., Kirschner, P., Clark, R.E. (2007). Why Minimally Guided Teaching Techniques Do Not Work: A Reply to Commentaries. Educational Psychologist. 42(2), pp 115 - 121.

Tutorials and the Significant Role of the TA's

It is unfortunate that many TA's are so busy with their research and course work that they often have minimal time to devote to tutorial preparation, whether in the material to be covered or teaching methods. A study by Koenig et al (2007) found that student performance gain in learning drastically improved in tutorials where students work in groups with TA's interaction using Socratic dialogue over tutorials where they work alone, or where they learn in a traditional lecture setting, or even in groups by themselves without other inputs from TA's. Student satisfaction of tutorials is clearly linked to the teaching performance of the TA's.

Interestingly though, when students were asked which style of tutorial did they prefer, more students indicate a traditional lecture style than Socratic group discussion with a TA, even though it is less effective. Perhaps the latter style moves the students out of their comfort zone a tad more than what they are used to and may seem to demand more work from them? In any case, this latter style seems to be more successful in moving students away from their initial misconceptions in the tutorials.

In order to implement such learning / teaching style in the tutorials, TA's will require weekly training to prepare for the tutorials. They have to work through the material and they need guidance on how to use Socratic dialogue with each tutorial topic.

Reference:

Koenig, K., Endorf, R., Braun, G. (15 May 2007). Effectiveness of Different Tutorial Recitation Teaching Methods and Its Implications for TA Training. The American Physical Society. Physical Review Special Topics - Physics Education Research. 3, 010104-1 to 010104-9.

Multiple Choice Questions

Here are some suggestions and results on studies of multiple choice questions (Haladyna and Downing, 1989).

1. Three-option questions are optimal for most examinees. Three-option questions provides the most information at the mid range of the score scale, two-option questions provides the most information for high-scoring examinees, and the four- and five-option questions provide the most information for low-scoring examinees.
   
2. Use question format rather than sentence completion format.
3. Use as many functional distractors as are feasible. Eliminate dysfunctional distractors.
4. Type K questions (i.e. where each option includes combination of answers such as A) 1, 2, and 3, B) 2 or 3, etc.) are more inefficient to construct, more laborious to read, make a heavier cognitive demand on the students. They can be used to measure complex, higher level thinking skills.
5. Place the keys to the questions equally in different positions throughout the exam.
6. Avoid incorrect grammar that may clue the examinees to the correct option.
7. Humor in the options lowers test anxiety.
8. Word the question positively and avoid negative phrasing.
9. Common student errors can be used to make up distractors.
10. "All of the above" option makes the questions more difficult and less discriminating.
11. Avoid, or use sparingly, the option "None of the above". Similar to "All of the above", the questions are more difficult, less discriminating, and test scores are less reliable.

References:

Haladyn, T. and Downing S. (1989). Validity of a Taxonomy of Multiple-Choice Item-Writing Rules. Applied Measurement in Education. 2(1), pp 51- 78.

Haladyn T. and Downing S. (1989). A Taxonomy of Multiple-Choice Item-Writing Rules. Applied Measurement in Education. 2(1), pp 37 - 50.