Asking Questions

When we pose questions to our students, they sequentially and iteratively go through four stages: comprehension, memory retrieval, judgment, and mapping (Conrad and Blair, 1996) (Tourangeau, 1984) (Oksenberg and Cannell, 1977). At any one of these stages, students may find it difficult to answer the questions due to the choice of words and the way the questions are asked. This may not because of their misconceptions of the subject matter but may indicate the questions need to be revised. Ding et al. summarized their results of validating clicker questions using interviews (2009).

In the comprehension stage, we want to make sure the students understand the problem accurately. In a think-aloud session, we may be able to see whether the students have misinterpreted the questions. Otherwise, this can be easily dismissed as a misconception that the students have.

In the memory retrieval stage, we want to make sure the students are accessing the relevant information to solve the problem. If any part of the question triggers the students that lead them in the wrong path, these questions can be seen as "trick" questions and are not testing the student learning.

In judgment, students need to perform the appropriate task to solve the problem given a correct retrieval of relevant information. If the questions are not clear about the context / conditions, the students may not be able reach a definite conclusion. In those cases, the questions need to be clarified.

In mapping, students need to correctly map the right answer to the right choice. Here, the choices provided must be clear and the students can make a definite choice.

Validating questions take time, and student interviews seem to be an effective way of helping instructors refine their questions. Teachers can also find out something about the student responses to the questions and see if there is a majority of them getting the questions wrong by examining the exam sores and their correlation with other data. Such forensic study may reveal how students interpret and think through the questions.

References:

Ding, L, Reay, N.W., Lee, A., Bao, L. (2009). Are We Asking the Right Questions? Validating Clicker Question Sequences by Student Interviews. American Journal of Physics. 77(7), pp 643 - 650.

Conrad F. and Blair, J. (1996). From Impressions to Data: Increasing the Objectivity of Cognitive Interviews. Proceedings of the Section on Survey Research Methods, American Statistical Association. (ASA, Alexandria, VA). p 1.

Tourangeau. R. (1984). Cognitive Science and Survey Methods. Cognitive Aspects of Survey Design: Building a Bridge Between Disciplines. Edited by T. Jabine, M. Straf, J. Tanur, and R. Tourangeau. (National Academics Press, Washington, DC). p 73.

Oksenberg, L. and Cannell, C. (1977). Some Factors Underlying the Validity of Response in Self-Report. Bull. I'Institut Int. Stati. 48, pp 325 - 346.