This question comes from trying to analyze my recent exam (exam I have given and corrected) statistically. I have a list of questions (20 in total) and each question is given a score from 0 to five, so total possible score is 100. Then I define pass/not pass via some cutoff (I give grades on the A-B-C-D-E-F scale, but here we only discuss F/notF). A logistic regression of pass on the 20 score variables obviously give a perfect fit, and the usual assumptions behind logistic regression are obviously not fulfilled (nothing special here about logistic regression , the question really are about regression modelling, not any specific kind of regression model).

But in reality what we have sampled here is the variables, not the students! The interesting analysis is about the influence on grades of specific question types, for the actual students we have, which are not in any sense a sample. But the variables is a sample from a huge population of potential questions! ("a sample", inn the sense that that the exam could easily have been made from some other, but similar, questions. It could even have been made by sampling from some question data base.) So what could we do in direction of a formal analysis? Some form of cross-validation seems natural, but we should really subsample variables, not objects (students).

Any good ideas, or good references? I tried google scholar, but couldn't find anything.


as answer to comments. I am mostly after seeing which question have most influence on the pass/fail decision. My exam is a calculus exam, with questions as (simple example) "Find the derivative of $f(x) = e^x \cos(x^2+3) $." (there are also longer questions, but divided in parts) counting this parts I have 20 sub-questions. This questions might be seen as a sample (at least conceptually) from a much larger set of potential questions. My students are really a fixed set, and I want to evaluate the exam as a test for this specific set of students. (so resampling by bootstrapping students do not seem natural). I sum the scores for each question and then decide the grade by some cutoff.

Now, if I use the scores on the 20 questions as variables, and fit a logistic regression model as $$ P(\text{fail}) = \frac1{1+\exp(-\beta_0 -\beta_1 Q_1 - \dots -\beta_20 Q_{20})} $$ I do obviously get a perfect fit! since the pass/fail decision was taken on the basis of those 20 variables, in a linear fashion. But still, the decision is not perfect, obviously, there are error sources. One source of error is things like the day form of the student. A very different kind of error source is the match between the questions and the knowledge of the student, in that with a different sample of 20 questions, the student might have got a different result. So the question is about how to model this situation (and then analyze according to that model).

  • $\begingroup$ Are you interested in evaluating the difficulty of particular questions? You may be interested in Item-Response-Theory. $\endgroup$
    – Andy W
    Jun 26, 2014 at 11:41
  • 3
    $\begingroup$ Although I can glean a vague sense of what you are trying to do from this post, I cannot really understand what it is saying. I do not understand the sense in which "variables" are "sampled," nor is it clear what a "note" is or what a "question type" refers to. Could you explain these things and make it plain to us, in non-technical language, what you are trying to accomplish? $\endgroup$
    – whuber
    Jun 26, 2014 at 14:04
  • 1
    $\begingroup$ Thanks! I will try to edit the question to make it clearer! $\endgroup$ Jun 26, 2014 at 16:58
  • 3
    $\begingroup$ Sounds as if, in addition to item response theory, you would benefit from looking into measurement theory, testing theory, perhaps even something as specific as generalizability theory. $\endgroup$
    – rolando2
    Jun 27, 2014 at 18:23
  • $\begingroup$ In many foreign languages, "note" is an equivalent word for "grade" in English. I wonder if that underlies some of the confusion. $\endgroup$
    – Silverfish
    Jan 10, 2015 at 13:23


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.