Designing Effective Questions

Good questions that engage students in discussions are essential in peer instruction, whether these questions are posed after a mini lecture (Mazur, 1997) or as the core of in-class instruction (Beatty et al, 2005). Every good question should try to achieve three goals: content goal (deals with the subject material that you want to illuminate, or the what's), process goal (deals with the cognitive skills you want students to exercise, or the how's), and metacognitive goal (deals with the beliefs about learning, thinking, the subject area, etc.).

Beatty et al. propose four tactics in designing good questions. They are listed here in the order that may be appropriate for an one hour lecture where usually four questions can be quite easily incorporated into the lesson:

1. Tactics for directing attention and raising awareness. Focusing student attention and increasing student motivation in learning are important aspects at the beginning of each lesson. Some of the ways to achieve this are to ensure the questions (or invention activities) have all nonessential material removed, provide opportunities for students to compare and contrast different cases, extending a familiar case to something different, setting a trap to show student misconceptions.
   
2. Tactics for promoting articulation discussion. Using unstated assumptions, deliberate ambiguity, questions with multiple possible answers, students can be challenged to discuss and articulate their thoughts, ideas, and to clarify the topic to be further presented.
   
3. Tactics for stimulating cognitive processes. The fundamental rule here is to ask questions that cannot be answered without exercising the desired habits of mind. Some of the methods include asking questions that require students to interpret representations, understand a process or algorithm (rather than just memorizing a formula), having students describe the meaning and to choose from a set of possible ways of solving a problem, comparing and making contrast of different cases, and having students identify the necessary information to continue in their learning.
   
4. Tactics for formative use of response data. By revealing other students' response to a question posed before via a response histogram, a follow up question can be used to drill further down into common student misconceptions and clarify the differences among them. Having students to explain their choice of answers also promote learning and discussion in the classroom.
   

References:

Beatty, I., Gerace, W., Leonard, W., Dufresne, R. (2005). Designing Effective Questions for Classroom Response System Teaching. American Association of Physics Teachers, American Journal of Physics. 74(1), pp 31 - 39.

Mazur, E. (1997). Peer Instruction: A User's Manual. Upper Saddle River, NJ: Prentice-Hall.

Multiple Choice Questions

Here are some suggestions and results on studies of multiple choice questions (Haladyna and Downing, 1989).

1. Three-option questions are optimal for most examinees. Three-option questions provides the most information at the mid range of the score scale, two-option questions provides the most information for high-scoring examinees, and the four- and five-option questions provide the most information for low-scoring examinees.
   
2. Use question format rather than sentence completion format.
3. Use as many functional distractors as are feasible. Eliminate dysfunctional distractors.
4. Type K questions (i.e. where each option includes combination of answers such as A) 1, 2, and 3, B) 2 or 3, etc.) are more inefficient to construct, more laborious to read, make a heavier cognitive demand on the students. They can be used to measure complex, higher level thinking skills.
5. Place the keys to the questions equally in different positions throughout the exam.
6. Avoid incorrect grammar that may clue the examinees to the correct option.
7. Humor in the options lowers test anxiety.
8. Word the question positively and avoid negative phrasing.
9. Common student errors can be used to make up distractors.
10. "All of the above" option makes the questions more difficult and less discriminating.
11. Avoid, or use sparingly, the option "None of the above". Similar to "All of the above", the questions are more difficult, less discriminating, and test scores are less reliable.

References:

Haladyn, T. and Downing S. (1989). Validity of a Taxonomy of Multiple-Choice Item-Writing Rules. Applied Measurement in Education. 2(1), pp 51- 78.

Haladyn T. and Downing S. (1989). A Taxonomy of Multiple-Choice Item-Writing Rules. Applied Measurement in Education. 2(1), pp 37 - 50.

Asking Questions

When we pose questions to our students, they sequentially and iteratively go through four stages: comprehension, memory retrieval, judgment, and mapping (Conrad and Blair, 1996) (Tourangeau, 1984) (Oksenberg and Cannell, 1977). At any one of these stages, students may find it difficult to answer the questions due to the choice of words and the way the questions are asked. This may not because of their misconceptions of the subject matter but may indicate the questions need to be revised. Ding et al. summarized their results of validating clicker questions using interviews (2009).

In the comprehension stage, we want to make sure the students understand the problem accurately. In a think-aloud session, we may be able to see whether the students have misinterpreted the questions. Otherwise, this can be easily dismissed as a misconception that the students have.

In the memory retrieval stage, we want to make sure the students are accessing the relevant information to solve the problem. If any part of the question triggers the students that lead them in the wrong path, these questions can be seen as "trick" questions and are not testing the student learning.

In judgment, students need to perform the appropriate task to solve the problem given a correct retrieval of relevant information. If the questions are not clear about the context / conditions, the students may not be able reach a definite conclusion. In those cases, the questions need to be clarified.

In mapping, students need to correctly map the right answer to the right choice. Here, the choices provided must be clear and the students can make a definite choice.

Validating questions take time, and student interviews seem to be an effective way of helping instructors refine their questions. Teachers can also find out something about the student responses to the questions and see if there is a majority of them getting the questions wrong by examining the exam sores and their correlation with other data. Such forensic study may reveal how students interpret and think through the questions.

References:

Ding, L, Reay, N.W., Lee, A., Bao, L. (2009). Are We Asking the Right Questions? Validating Clicker Question Sequences by Student Interviews. American Journal of Physics. 77(7), pp 643 - 650.

Conrad F. and Blair, J. (1996). From Impressions to Data: Increasing the Objectivity of Cognitive Interviews. Proceedings of the Section on Survey Research Methods, American Statistical Association. (ASA, Alexandria, VA). p 1.

Tourangeau. R. (1984). Cognitive Science and Survey Methods. Cognitive Aspects of Survey Design: Building a Bridge Between Disciplines. Edited by T. Jabine, M. Straf, J. Tanur, and R. Tourangeau. (National Academics Press, Washington, DC). p 73.

Oksenberg, L. and Cannell, C. (1977). Some Factors Underlying the Validity of Response in Self-Report. Bull. I'Institut Int. Stati. 48, pp 325 - 346.