Are Clickers Really Effective in Improving Student Performance?

A survey of students in six biology courses showed that students not only have favorable opinions about the use of student response systems (or clickers), but clicker usages also increase student learning (Preszler et al, 2007). Prior to this, Judson and Sawada (2002) have shown that students consistently show positive evaluations of using clickers in class for three decades, but there has been no consistent demonstration of learning improvement until this study.

81% of the students in the study agreed that using clickers increased their interest in their course. 71% of students agreed that clickers made it more likely for them to attend class. 70% agreed that clickers improved their understanding of course material. Most important, there was a significant linear increase in exam scores across all three levels of clicker usage frequency per class (low - 0 to 2, medium - 2 to 3, high - 4 to 6). That is high clicker usage results in mean student grades greater than medium clicker usage, and medium clicker usage results in greater mean student grades than low clicker usage.

Methodology: The study by Preszler et al. first analyzed whether the course grade distribution is similar among the six courses (using stepwise chi-square analysis), and then the students' opinions of clicker usage is analyzed to see if they differ by course grades (again using stepwise chi-square analysis). Courses that are significantly different are not included in the analysis to preserve as much consistency among the courses as possible. Clicker usage frequency in the courses follows a Latin square design to maintain an overall similar equivalent number of clicker questions used over the testing period to avoid biasing variation.

Reference:

Judson, E., and Sawada, D. (2002). Learning from the past and present: electronic response systems in college lecture halls. Journal of Computers in Mathematics and Science Teaching. 21(2), pp 167 - 181.

Preszler, R., Dawe, A., Shuster, C., and Shuster, M. (Spring 2007). Assessment of the Effects of Student Response Systems on Student Learning and Attitudes over a Broad Range of Biology Courses. Life Sciences Education. Vol 6, pp 29 - 41.

Goal-Free Problems

Asking a student to find the solution to a problem creates high cognitive load that she may end up making a lot of mistakes. This is especially true when the problem requires many sub problems to be solved, and students tend to make many more errors in these sub goal stages than at the final goal stage. This effect is called the stage effect.

An alternative to help students learn problem solving is to ask them to find the value of as many unknowns as possible, rather than finding a value for a specific goal. As an example, given a programming assignment, students are asked what are the unknowns rather than asking them to create a final program. This can be open ended and the students may go off on a tangent if not properly guided. Of course, no one would hire these students if all they can do is to come up with unknowns(!), but this strategy can be used as a scaffolding device to help students connect what they already know and what we want them to know. New knowledge can be built up as the unknowns are identified, and how these unknowns are related to what has already been learned.

Reference:

Ayres, P. (1993). Why Goal-Free Problems Can Facilitate Learning. Contemporary Educational Psychology. 18, pp 376 - 381.

Value of Praising Your Students / Kids

We all love praises ... for a job well done, for academic achievements, for beauty, .. but what do they do to us? Well we all know that they inflate our ego's, but unknowingly, they may have more damaging effects than we think!

Research has shown that people tend to give up if they realize that their lack of performance is due to a lack of ability, whereas people tend to continue trying if they realize that it is due to a lack of effort. It should be clarified here that "ability" refers to something that is fixed, whether it is true or not. Some people believe that intelligence is fixed and cannot be changed. Others may believe that playing a musical instrument is an innate ability rather than learned. These are often referred to as fixed or growth mindset. Students with fixed mind set are concerned about looking smart with little regard of learning. Students with a growth mind set are more concerned about learning than getting good grades.

Dweck (2007) found out that praising someone's intelligence encourages a fixed mind set more often than praising them for their effort. The underlying belief system is that we tend to think that intelligence is fixed. Research has also shown that those who were praised for their intelligence tend to shy away from challenging assignments, and this is far more often than those who were applauded for their effort.

Children who are praised for their intelligence also tend to pursue performance goal which means that their primary motivation is to continue to prove that they are intelligent by the rewards or recognition they can get. This can have negative consequences in that they are likely to sacrifice potential learning opportunities if these opportunities have an element of risk of making errors and do not ensure immediate good performance. Children who are praised for their effort prefer a learning goal that emphasizes the mastery of new and challenging material.

Children praised for intelligence were less likely to want to persist on problems than children praised for effort (Mueller and Dweck, 1998). It has also been shown that children praised for intelligence also enjoyed the tasks assigned to them less than children praised for effort. In another experiment, children praised for intelligence perform worse than children praised for effort after encountering failures and setbacks.

In yet another study, Nussbaum and Dweck (2008) show that people who have a fixed mind set of intelligence (also called entity condition) tend to repair their self esteem "defensively" by comparing themselves with competitors of equal or lower abilities after they encounter failures, whereas people who have a growth mind set of intelligence (also called incremental condition) tend to repair their self esteem by trying to engage in remedial learning and comparing themselves with competitors of higher abilities.

References:

Dweck, Carol. (November 28, 2007). The Secret to Raising Smart Kids. Scientific American Mind.

Mueller, C. and Dweck, C. (1998). Praise for Intelligence Can Undermine Children's Motivation and Performance. Journal of Personality and Social Psychology. 75(1), pp 33 - 52.

Nussbaum, D. and Dweck, C. (May 2008). Defensiveness Versus Remediation: Self-Theories and Modes of Self-Esteem Maintenance. Personality and Social Psychology Bulletin. 34(5), pp 599 - 612.

Framing

Framing is a construct developed in anthropology and linguistics to describe how an individual or group forms a sense of "what is it that's going on here?". We frame an event, utterance, or situation by interpreting it based on previous experience. E.g. when we see someone running like a madman on the street, we may interpret that as a fugitive on the run, and may expect someone else is chasing after him. Students may look at an exam question and quickly associate the same question with a previous exercise problem she has seen before.

Epistemological framing refers to the way learners form a sense of what is taking place with respect to knowledge, e.g. what past experience or knowledge is relevant to complete an assignment. Social framing refers to the way people form a sense of what to expect of each other, and of themselves in a social setting, e.g. what students expect from each other in a group project. Social framing can be observed through people's behaviors. Epistemological framing can be deduced through student learning assessments and their problem solving skills.

Based on the idea of social and epistemological framing, Scherr and Hammer (2009) studied how student interact with each other in physics tutorials. They coded student behaviors based on whether they work alone, discuss with each other, discuss with the TA, or just social, and correlate with student thinking, and their epistemological framing. They show that the behavioral cluster are evidence of student epistemologies. In particular, sitting up, speaking clearly, and gesturing frequently are evidence of novel reasoning and mutually constructed understanding.

Reference:

Scherr, R. and Hammer, D. (2009). Student Behavior and Epistemological Framing: Examples From Collaborative Active-Learning Activities in Physics. Cognition and Instruction, 27(2), pp 147 - 174.

Designing Effective Questions

Good questions that engage students in discussions are essential in peer instruction, whether these questions are posed after a mini lecture (Mazur, 1997) or as the core of in-class instruction (Beatty et al, 2005). Every good question should try to achieve three goals: content goal (deals with the subject material that you want to illuminate, or the what's), process goal (deals with the cognitive skills you want students to exercise, or the how's), and metacognitive goal (deals with the beliefs about learning, thinking, the subject area, etc.).

Beatty et al. propose four tactics in designing good questions. They are listed here in the order that may be appropriate for an one hour lecture where usually four questions can be quite easily incorporated into the lesson:

1. Tactics for directing attention and raising awareness. Focusing student attention and increasing student motivation in learning are important aspects at the beginning of each lesson. Some of the ways to achieve this are to ensure the questions (or invention activities) have all nonessential material removed, provide opportunities for students to compare and contrast different cases, extending a familiar case to something different, setting a trap to show student misconceptions.
   
2. Tactics for promoting articulation discussion. Using unstated assumptions, deliberate ambiguity, questions with multiple possible answers, students can be challenged to discuss and articulate their thoughts, ideas, and to clarify the topic to be further presented.
   
3. Tactics for stimulating cognitive processes. The fundamental rule here is to ask questions that cannot be answered without exercising the desired habits of mind. Some of the methods include asking questions that require students to interpret representations, understand a process or algorithm (rather than just memorizing a formula), having students describe the meaning and to choose from a set of possible ways of solving a problem, comparing and making contrast of different cases, and having students identify the necessary information to continue in their learning.
   
4. Tactics for formative use of response data. By revealing other students' response to a question posed before via a response histogram, a follow up question can be used to drill further down into common student misconceptions and clarify the differences among them. Having students to explain their choice of answers also promote learning and discussion in the classroom.
   

References:

Beatty, I., Gerace, W., Leonard, W., Dufresne, R. (2005). Designing Effective Questions for Classroom Response System Teaching. American Association of Physics Teachers, American Journal of Physics. 74(1), pp 31 - 39.

Mazur, E. (1997). Peer Instruction: A User's Manual. Upper Saddle River, NJ: Prentice-Hall.

Prospective Adaptation

One of the many goals of an educator is to prepare their students to adapt what they have learned in new situations. There can be two types of adaptation: fault-driven adaptation (which are reactions to a difficult situation), and prospective adaptation (which are proactive reformulations of one's knowledge or environment prior to encountering a new problem of situation). Martin and Schwartz (2009) show that graduate students uniformly make prospective adaptations to create meaningful representations of available information much more often than undergraduate students before diagnosing a problem, even though this may cost them some start up time. Undergraduate students who do not have continuous access to reference material tend to create more meaningful representations than students who have continuous access. The long term benefit for the graduate students in creating meaningful representation through prospective adaptation is that they complete a new diagnostic task much quicker than others with no meaningful representation.

In Computer Science, as in many other Science disciplines, students are not usually given the time or opportunity to step back, reflect, and retool one's knowledge. In first year programming, students are taught to solve computing problems through a systematic and methodical way while they may not understand why "hacking" is not suitable. But rather than short circuiting this process of prospective adaptation which the students should work through themselves, perhaps the students should be allowed to hack their code, and then asked to step back to rethink what other ways of problem solving is more appropriate when given a more complex problem. Instead, most often, algorithmic or procedural formulations are provided and students often try to memorize these solutions, hoping that these are sufficient for any new problems they will encounter. A learning goal should be included in each course where students are expected to engage in prospective adaptation to create new representations and ways of integrating new ideas with their knowledge base. This can be in the form of creating concept maps, writing up summaries of new knowledge and its connections to other areas (e.g. through a blog), designing new procedures or methods in solving problems, or engaging in invention activities (which encourage risk taking at little cost).

Reference:

Martin, L. and Schwartz, D. (2009). Prospective Adaptation in the Use of External Representations. Cognition and Instruction, 27(04), pp 370 - 400.

Peerwise

Peerwise is a collaborative web-based system that allows students to create and evaluate a test bank of multiple choice questions. The pedagogical motivation behind this system is that students can learn better if they go through a process of self-reflection (meta-cognition), identify / synthesize / evaluate (higher levels of Bloom taxonomy), and articulate the subtleties in a concise format. Denny et al. (2010) show that students who were most active using the system improved their rank in the class relative to their peers who were less active. This is measured by using the students' final course grade from the previous course as a baseline for their initial class rank, and comparing with their final course grade in the course that involves the use of Peerwise.

Reference:

Denny, P., Hanks, B., Simon, B. (2010). PeerWise: Replication Study of a Student-Collaborative Self-Testing Web Service in a U.S. Setting. SIGCSE 2010, March 10-13.

Learner's Styles, Aptitudes, Personalities .. do they make a difference?

Learning styles refer to the different ways different people learn information (e.g. visual / audio learners). Learning aptitudes refer to how different people learn in different learning environment structure (e.g. how students learn in highly structured or less structured learning environments). Learner personalities refer to the learner's belief whether his or her successes or failures are a consequence of internal or external factors (e.g. whether students believe their success and failures are a consequence of internal or external factors). Pashler et al. (2009) report that there are inconsistent and insufficient evidences that learning will be effective if instructions are provided in the mode that match learner's styles / attributes / personalities. This does not mean that learners do not have preferences, but in the particular type of evidence that Pashler et al. are looking for, that according to them would be "credible validation of learning-styles-based instruction", such evidence is missing.

The lack of evidence also does not mean that instructors should just stick to one mode of teaching. Students benefit from different representations of information, whether it be verbal, visual, analytical, lecture-based, inductive / deductive reasoning, etc., and that students should not pigeon-holed themselves in learning from any one or two particular styles.

Reference:

Pashler, H., McDaniel, M., Rohrer, D., and Bjork, R. (2009). Learning Styles, Concepts and Evidence. Psychological Science in The Public Interest. 9(3), pp 105- 119.

Student Self-Explanation

Student self-explanation of material they just read has been shown to be effective in producing robust learning gains in a number of disciplines. However, past research results have not been clear whether performance gain is due to student simply paying attention to explanation generated by the instructors, or explanation generated by the students themselves. One research has shown that explanation is more effective when the students generate it rather than simply paying attention to instructor generated explanations (Brown and Kane, 1988), while in another case, the reverse is true (Lovett, 1992). Most recently, Hausmann and Vanlehn (2007) show that generating self-explanation while students attempted solving problems and studying examples is more effective in normal as well as robust learning (which means knowledge is retained over a significant period of time and demonstrated in far transfer of problem solving) than students who comprehended and paraphrased explanations generated by the instructors.

Self-explanation, coupled with learning by examples, can be very effective in student learning. Learning by examples has a lower cognitive load than learning by doing or solving problems, based on cognitive load theory. Thus comparing students who learn by doing a number of questions with those who learn by working through a number of examples, the cognitive load in the latter is much lower, and this affords the students the capacity to come up with general solution principles through self-explanation to improve their effectiveness in learning.

A related theme is that students who self-monitor their learning and comprehension in addition to self-explain the material they learned are better problem solvers than those who don't. By self-monitoring, this means that the students keep track of what they know and what they don't know, what are the parameters and data provided by the problems they are trying to solve, what needs to be solved, how the problems relate to the examples they have worked through having specific goals such as looking for solution methods rather than equations, formulas, similar contexts, etc.

References:

Brown, A.L. and Kane, M.J. (1988). Preschool Children Can Learn to Transfer: Learning to Learn and Learning from example. Cognitive Psychology. 20(4), pp 493 - 523.

Hausmann, R.G.M. and Vanlehn, K. (2007). Explaining Self-Explaining: A Contrast Between Content and Generation. In R. Luckin, K.R. Koedinger, and J. Greer (Eds). Proceedings of Artificial Intelligence in Education (2007). Amsterdam, The Netherlands: IOS Press.

Lovett, M.C. (1992). Learning by Problem Solving versus by Examples: The Benefits of Generating and Receiving Information. Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society, Hillsdale, NJ: Erlbaum, pp 956 - 961.

Optimized University

Carl Wieman put together a "think piece" on a new model for post-secondary education, which he called the Optimized University. Here are some of the highlights:

• The Optimized University will focus on the desired student education outcomes rather than number of courses / credits students need to graduate with. There will be a switch in focus from processes to outcomes.
• The instructor's role will primarily be an educational designer who continually assesses student's development with the assistance of technology and provides targeted feedback and challenges to the students to optimize their learning rather than simply a one-way transference of knowledge to students.
• Clearly delineated educational goals will be created by relevant faculty in consultation with other stakeholders such as industry, educational systems, and government.
• IT will be used to accurately diagnose student preparation, conceptual knowledge, beliefs, and epistemologies. IT will also be used for new teaching methods (interactive simulations, intelligent tutors, sophisticated diagnostic capabilities, clickers), improved class organization and management systems, archiving systems for educational materials and data, deployment of new modes of presenting material and enhanced communication by linking students with each other and faulty.
• The Optimized University will have sophisticated pedagogical content knowledge - knowledge on how the content and skills are best learned, common student difficulties, approaches most effective in helping students overcome those difficulties, and how to motivate students to master the subject.
• Validated assessments of desired deep understanding of material rather than a simple memorization of facts and problem solving recipes will be in place.
• Technology will be used to make classes more intellectually engaging and educationally effective. Research has shown that there have been demonstrations of classes of 200 or more achieving very good learning gains using clickers and peer instruction in the lectures, computer graded homework systems, student-student collaboration (on / off line), extensive course webpages, and survey systems.
• Carefully constructed diagnostic exams will be used to assess student preparedness and to reduce large hidden cost in instructor's time to provide the unprepared students with extra assistance and in dealing with the repercussions of failing students.
• Student support will range from peer support and intelligent tutoring system, to trained undergraduate and graduate TA, to the expertise available from the faculty.
• Students will have authentic research experience upon graduation.

Reference:

Wieman, Carl. (n.d.) A New Model for Post-Secondary Education, the Optimized University. Retrieived on January 2, 2010, from here.