Item Response Theory

How do we (as instructors) decide whether a test is "hard" or "easy"?  Most of us will answer something along the line .. "it all depends".  I find this observation which Hambleton et al. make of the common responses to this question interesting: "Whether an item [or test] is hard or easy depends on the ability of the examinees being measured, and the ability of the examinees depends on whether the test items are hard or easy!"  Not very helpful, isn't it? Item Response Theory is a body of theory which applies mathematical models to analyze student scores of individual questions from a test to facilitate comparison of the difficulty level of the questions and their capabilities to differentiate student abilities.  It is based on two basic postulates: 1) the performance of an examinee can be predicted by a set of factors called traits (or abilities), 2) the relationship between examinees' item performance and the set of traits can be described by an item characteristic function or item characteristic curve (ICC) like the one in the graph above.  The x axis is the trait or ability score, and the y axis is the probability of the examinee with certain trait or ability score to obtain the correct answer.  As the ability of an examinee increases, so does the probability of a correct response to an item.

Each item in a test has its own ICC and the ICC is the basic building block of IRT.  The steepness of the graph shows how well the item can differentiate examinees with low and high abilities.  A flat curve is a poor indicator while a steep curve, like the one shown above, is a good indicator.  If several ICC's are plotted in the same graph for the corresponding test items with the same shape, the curves on the left (or top) correspond to the easier items than those on the right (or bottom).

By analyzing the examinees' scores of each item from an exam using IRT software, one can have an idea 1) which questions are good indicators of assessing student abilities or not, and 2) objectively respond to which questions are "easy" or "hard".

Reference:

Baker, F. The Basics of Item Response Theory. Available online here.

Graph is taken from http://echo.edres.org:8080/irt/ where one can also find a great deal of information on IRT.

Hambleton, R., Swaminathan, H., Rogers, H.  Fundamentals of Item Response Theory. Newbury Park: Sage Publications. 1991.