Is there a statistical reason why item analysis/response theory isn't more widely applied? For instance, if a teacher gives a 25 question multiple choice test and finds that 10 questions were answered correctly by everyone, 10 questions were answered by a really low fraction (say 10%) and the remaining 5 were answered by roughly 50% of people. Doesn't it make sense to reweight the scores so that hard questions are given more weight?

And yet, in the real world tests almost always have all questions weighted equally. Why?

The below link discusses discrimination indices and other measures of difficulties for choosing which questions are best: http://fcit.usf.edu/assessment/selected/responsec.html

It seems though that the method of figuring out the discrimination index of questions is only used in a forward-looking way (eg., if a question doesn't discriminate well, toss it). Why aren't tests re-weighted for the current population?


(You asked whether there is a statistical reason: I doubt it, but I'm guessing about other reasons.) Would there be cries of "moving the goalpost"? Students usually like to know when taking a test just how much each item is worth. They might be justified in complaining upon seeing, for example, that some of their hard-worked answers didn't end up counting much.

Many teachers and professors use unsystematic, subjective criteria for scoring tests. But those who do use systems are probably wary about opening those systems up to specific criticism -- something they can largely avoid if hiding behind more subjective approaches. That might explain why item analysis and IRT are not used more widely than they are.

  • 1
    $\begingroup$ +1, I expect many teachers and professors are unfamiliar with IRT, as well. $\endgroup$ – gung - Reinstate Monica Nov 25 '11 at 4:02
  • $\begingroup$ Absolutely! Another non-statistical reason. I find that teacher education in this country is such that many who teach receive little or no training in assessment. And then there are researchers and test-designers trained in assessment who have little or no training in instruction methods. $\endgroup$ – rolando2 Nov 25 '11 at 14:55
  • 1
    $\begingroup$ @rolando2 -- I'm ashamed as an economist that the thought of ex-ante or while-taking-the-test incentives didn't occur to me. If test takers are dealing with a moving target and don't know how to rationally spend their time and effort answering each question, that could probably have some really perverse effects! $\endgroup$ – d_a_c321 Nov 26 '11 at 21:04
  • $\begingroup$ I see how it could disgruntle them, but what kind of "perverse" effects are you thinking of? $\endgroup$ – rolando2 Nov 30 '11 at 10:00

A first argument has do with transparency. @rolando2 has already made this point. The students want to know ex-ante how much each item is worth.

A second argument is that the weights do not only reflect the degree of difficulty of a question, but also the degree of importance the instructor attaches to a question. Indeed, the aim of an exam is testing and certifying knowledge and competencies. As such, the weights attributed to different questions and items have to be set beforehand by the teacher. You should not forget that "all models are wrong, and only some are useful". In this case one can have some doubts on the usefulness.

This being said, I think that (more or less fancy) statistical analysis could come in ex-post, for the analysis of the results. There it can yield some interesting insights. Now, if this is done and to what degree it is done, depends certainly on the statistical skills of the teacher.

  • 2
    $\begingroup$ Absolutely! As an instructor, what I'm looking for is not a relative ranking of the students -- but rather an absolute measure of comprehension and skills. I would be perfectly happen (delighted, even), if it turned out that everyone in the class was measured as having 100% comprehension of the subject I'm teaching. So, the weights on questions are chosen to reflect their importance and how much weight they should have in an overall assessment of understanding of the subject. $\endgroup$ – D.W. Nov 27 '11 at 21:44

I wanted to make a clarification regarding the original question. In item response theory, the discrimination (i.e. item slope or factor loading) is not indicative of difficulty. Using a model that allows for varying discrimination for each item is effectively weighting them according to their estimated correlation to the latent variable, not by their difficulty.

In other words, a more difficult item could be weighted down if it estimated to be fairly uncorrelated with the dimension of interest and vice versa, an easier item could be weighted up if is estimated to be highly correlated.

I agree with previous answers that point to (a) the lack of awareness of item response methods among practitioners, (b) the fact that using these models require some technical expertise even if one is aware of their advantages (specially the ability of evaluating the fit of the measurement model), (c) the student's expectations as pointed out by @rolando2, and last but not least (d) the theoretical considerations that instructors may have for weighting different items differently. However, I did want to mention that:

  1. Not all item response theory models allow for variation of the discrimination parameter, where the Rasch model is probably the best known example of a model where the discriminations across items are held constant. Under the Rasch family of models, the sum score is a sufficient statistic for the item response score, therefore, there will be no difference in the order of the respondents, and the only practical differences will be appreciated if the 'distances' between the score groups are considered.

  2. There are researchers that defend the use of classical test theory (which relies on the traditional use of sum scores or average correct) for both theoretical and empirical reasons. Perhaps the most used argument is the fact that scores generated under item response theory are effectively very similar to the ones produced under classical test theory. See for example the work by Xu & Stone (2011), Using IRT Trait Estimates Versus Summated Scores in Predicting Outcomes, Educational and Psychological Measurement, where they report correlations over .97 under a wide array of conditions.

  • $\begingroup$ (+1) The first paragraph is very well put. About point 2, I somehow missed this article, so thanks for sharing! Of note, correlations of comparable magnitude would be observed with traditional factor analysis models (where loadings mimic the discrimination parameter -- under the CTT framework, it would be the point-biserial correlation coefficient) provided item difficulties are uniformly distributed on the latent trait (i.e., no too difficult nor too easy items). $\endgroup$ – chl Jan 2 '12 at 22:31

Shouldn't a student's score be based on what they know and answer on the test rather than what everyone else in the class does?

If you gave the same test 2 different years and you had 2 students (1 in each) who answered the exact same questions correctly (without cheating), does it really make sense that they would received different grades based on how much the other students in their class studied?

And personally, I don't want to give any students motivation to sabatoge their class mates in place of learning the material themselves.

IRT can give some insight into the test, but I would not use it to actively weight the scores.

When I think of weights, I think that someone should get more points for getting a hard question correct, but they should lose more points for getting an easy question wrong. Combine those and you still end up with equal weighting. Or I actually try to weight based on time or effort needed to answer the question, so that someone who answers the questions in a different order does not have an advantage on a timed test.

  • $\begingroup$ Large testing organisations that use IRT also have to worry about consistency between offering of the test. Consistency of the latent variable(s) is important but is attainable. $\endgroup$ – D Coetzee Aug 13 '12 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.