31
$\begingroup$

Here is a simple statistics question I was given. I'm not really sure I understand it.

X = the number of aquired points in an exam (multiple choice and a right answer is one point). Is X binomial distributed?

The professor's answer was:

Yes, because there is only right or wrong answers.

My answer:

No, because each question has a different "success-probability" p. As I did understand a binomial distribution is just a series of Bernoulli-experiments, which each have a simple outcome (success or failure) with a given success-probability p (and all are "identical" regarding p). E.g., Flipping a (fair) coin 100 times, this is 100 Bernoulli-experiments and all have p=0.5 . But here the questions have different kinds of p right?

Improve this question
$\endgroup$
10
  • 14
    $\begingroup$ +1 Even more to the point: unless this is a strange exam indeed, the responses to the questions will be strongly correlated. If $X$ is the total score for an individual, this will preclude a Binomial distribution. Could it be possible the question is operating under a "null hypothesis" assumption in which all examinees are independently and randomly guessing all answers? $\endgroup$
    – whuber ♦
    Apr 19, 2018 at 18:09
  • 2
    $\begingroup$ How paradoxical, I would have at least lobbied for partial credit on this, but the "answer" seems to reflect a disinclination to award it :) (I think you're right here). $\endgroup$
    – AdamO
    Apr 19, 2018 at 18:17
  • 1
    $\begingroup$ Yes, thank you :D, i think its more a Poisson binomial distribution (if anything) $\endgroup$
    – Paul
    Apr 19, 2018 at 18:17
  • 2
    $\begingroup$ @Zahava See stats.stackexchange.com/search?q=poisson+binomial. $\endgroup$
    – whuber ♦
    Apr 19, 2018 at 18:28
  • 2
    $\begingroup$ I agree with everyone that the question was poor, but there's a framing issue here. If this is an elementary course and it's a short-answer format (so that you have a chance to explain your reasoning), I would say the best answer is probably "yes (assuming independence and equal difficulty for each question)"; that would signal to the professor that (1) you understand the limitations of the question and (2) you're not trying to be a smart-ass. $\endgroup$
    – Ben Bolker
    Apr 20, 2018 at 22:07

4 Answers 4

Reset to default
26
$\begingroup$

I would agree with your answer. Usually this kind of data would nowadays be modeled with some kind of Item Response Theory model. For example, if you used the Rasch model, then the binary answer $X_{ni}$ would be modeled as

$$ \Pr \{X_{ni}=1\} =\frac{e^{{\beta_n} - {\delta_i}}}{1 + e^{{\beta_n} - {\delta_i}}} $$

where $\beta_n$ can be thought as $n$-th persons ability and $\delta_i$ as $i$-th question difficulty. So the model enables you to catch the fact that different persons vary in abilities and questions vary in difficulty, and this is the simplest of the IRT models.

Your professors answer assumes that all questions have same probability of "success" and are independent, since binomial is a distribution of a sum of $n$ i.i.d. Bernoulli trials. It ignores the two kinds of dependencies described above.

As noticed in the comments, if you looked at the distribution of answers of a particular person (so you don't have to care about between-person variability), or answers of different people on the same item (so there is no between-item variability), then the distribution would be Poisson-binomial, i.e. the distribution of the sum of $n$ non-i.i.d. Bernoulli trials. The distribution could be approximated with binomial, or Poisson, but that's all. Otherwise you're making the i.i.d. assumption.

Even under "null" assumption about guessing, this assumes that there is no guessing patterns, so people do not differ in how they guess and items do not differ in how they are guessed--so the guessing is purely random.

Improve this answer
$\endgroup$
3
  • $\begingroup$ That makes sense! Although i guess you could compute the probability of the success probability of a question but the "persons ability" sounds difficult :) Another idea that i had is to model this as a sum of bernulli distributions? E.g. lets say there are 2 question, therefore 2 success-probabilities p1 and p2. Analogously two variables X1 and X2 counting (so 2 bernulli-experiments). Then for example the probability of getting one total score of 1 is P(X1=1)*P(X2=0)+P(X1=0)*P(X2=1)=p1(1-p2)+(p1-1)p2. Does that sound reasonable? $\endgroup$
    – Paul
    Apr 19, 2018 at 20:47
  • 2
    $\begingroup$ @Paul sum of two Bernoulli's with different p's is Poisson-binomial $\endgroup$
    – Tim ♦
    Apr 19, 2018 at 20:49
  • 4
    $\begingroup$ The "null" assumption is basically a spherical-cow thing, you can always quibble about exactly how spherical the cow is. $\endgroup$
    – Hong Ooi
    Apr 20, 2018 at 2:36
5
$\begingroup$

The answer to this problem depends on the framing of the question and when information is gained. Overall, I tend to agree with the professor but think the explanation of his/her answer is poor and the professor's question should include more information up front.

If you consider an infinite number of potential exam questions, and you draw one at random for question 1, draw one at random for question 2, etc. Then going into the exam:

  1. Each question has two outcomes (right or wrong)
  2. There are a fixed number of trials (questions)
  3. Each trial could be considered independent (going into question two, your probability $p$ of getting it right is the same as when going into question one)

Under this framework, the assumptions of a binomial experiment are met.

Alas, ill-proposed statistical problems are very common in practice, not just on exams. I wouldn't hesitate to defend your rationale to your professor.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Jea i guess that is right too. The question is just "bad", since you could argue both ways, since so little information is given. But i just was very unhappy with the given answer of my professor. $\endgroup$
    – Paul
    Apr 19, 2018 at 20:57
  • 4
    $\begingroup$ @Paul, it's actually pretty hard to write good statistical questions. I know I've flubbed it on many occasions. $\endgroup$
    – gung - Reinstate Monica
    Apr 19, 2018 at 21:01
  • 1
    $\begingroup$ If you consider an infinite number of potential exam questions, and you draw one at random for question 1, draw one at random for question 2, etc. -- I think you should make explicit the assumption that exam questions are drawn independently from the pool of potential questions. It would be more realistic for them to be correlated: if question 1 is easy, it is likely that you are being given an easy exam and that question 2 will be easy. $\endgroup$
    – Adrian
    Apr 20, 2018 at 5:38
1
$\begingroup$

If there are n questions, and I can answer any one question correctly with probability p, and there is enough time to attempt answering all questions, and I did 100 of these tests, then my scores would be normal distributed with a mean of np.

But it's not me repeating the test 100 times, it's 100 different candidates doing one test, each with his own probability p. The distribution of these p's will be the overriding factor. You might have a test where p = 0.9 if you studied the subject well, p = 0.1 if you didn't, with very few people between 0.1 and 0.9. The distribution of points will have very strong maxima at 0.1n and 0.9 n and will be nowhere near normal distribution.

On the other hand, there are tests where everybody can answer any question, but take different amounts of time, so some will answer all n questions, and others will answer fewer because they run out of time. If we can assume that the speed of the candidates is normal distributed, then the points will be close to normal distributed.

But many tests will contain some very hard and some very easy questions, intentionally so that we can distinguish between the best candidates (who will answer all questions up to some degree of difficulty) and the worst candidates (who will only be able to answer very simple questions). This would change the distribution of points quite strongly.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ The normal distribution that you describe in here is normal approximation of binomial. Obviously the sum of zeros and ones wouldn't be continuous and range between $-\infty$ and $\infty$ $\endgroup$
    – Tim ♦
    Apr 20, 2018 at 4:27
  • 2
    $\begingroup$ @Tim Despite the unnecessary reliance on normal distributions and the mystery of taking 100 tests, this answer has merit in attempting to demonstrate how a particular case can lead to an obviously non-binomial distribution. As such it could be a valuable contribution to the answers if these technical issues were addressed. $\endgroup$
    – whuber ♦
    Apr 20, 2018 at 16:38
1
$\begingroup$

By definition, a binomial distribution is a set of $n$ independent and identically distributed Bernoulli trials. In the case of a multiple choice exam, each of the $n$ questions would be one of the Bernoulli trials.

The issue here arises because we can't reasonably assume that the $n$ questions:

  • Are identically distributed. As you said, the probability a student knows the answer to question $1$ is almost certainly not going to be the same as the probability they know the answer question $2$, and so on.
  • Are independent. Many exams ask questions that are built upon the answers to the previous question(s). Who's to say for sure that that wouldn't happen on the exam in this question? There are other factors that could make answers to exam questions not independent of one another, but I think this one is the most intuitively obvious.

I have seen questions in Statistics classes that model exam questions as binomials, but they are framed something along the lines of:

What probability distribution would model the number of questions answered correctly on a multiple choice exam where every question has four choices, and the student taking the exam is guessing every answer at random?

In this scenario, of course it would be represented as a binomial distribution with $p= \frac{1}{4}$ .

Improve this answer
$\endgroup$
1
  • $\begingroup$ There's nothing the matter with your facts, but the logic is incorrect: it doesn't suffice to demonstrate that some assumptions may not hold, because (logically) the distribution could still be binomial in any case. You also need to demonstrate that these assumptions can fail in ways that cause the score distribution definitely to be non-binomial. $\endgroup$
    – whuber ♦
    Apr 20, 2018 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.