Demonstrating that some exam questions are easier than others

Suppose there is an exam with 10 questions.

Each question is worth 1 mark, and can be answered either correctly or incorrectly, thereby being either awarded the mark or not.

100 people take this exam, and it's clear from the results that there are some questions that almost everyone gets right, and others that are almost universally answered wrong.

The total count of all correct answers for each question, in a table, looks like this:

+------------------+----+----+----+----+----+----+----+----+----+-----+
|                  | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 |
+------------------+----+----+----+----+----+----+----+----+----+-----+
| Correct answers  | 60 | 50 | 40 |  5 | 90 | 50 | 35 | 30 | 80 |  10 |
+------------------+----+----+----+----+----+----+----+----+----+-----+


It's clear that 90% of the 100 participants got Q5 right, but only 5% got Q4 right. You can see that some questions must be easier than others.

So my question is:

What's the clearest and cleanest statistical test to provide support for the 'some questions are easier than others' hypothesis in this case?

Perhaps you could assume that, given no question difficulty bias, the correct answers should be evenly spread between the questions. In this case, there were 450 correct answers, so you could compare expected versus actual, and see if there's a significant difference:

+----------+----+----+----+----+----+----+----+----+----+-----+
|          | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 |
+----------+----+----+----+----+----+----+----+----+----+-----+
| Expected | 45 | 45 | 45 | 45 | 45 | 45 | 45 | 45 | 45 |  45 |
| Actual   | 60 | 50 | 40 |  5 | 90 | 50 | 35 | 30 | 80 |  10 |
+----------+----+----+----+----+----+----+----+----+----+-----+


Perhaps you could use chi-squared to see if there is a significant difference between the expected 'even spread' and the actual results. But that seems clumsy. Is there a better way to do this?

Also, it would be great to see with some confidence which questions were easier, and some measure of 'how much easier'. Maybe that could be seen in chi-squared residuals, but I can't help feeling there's a better way altogether.

• A standard model for this problem is called "IRT" or "item response theory." – Sycorax Jul 5 '18 at 20:38
• @Sycorax: Thanks for the comment. I took a look at Wikipedia on IRT, and it seems quite a complicated model - I'm afraid I can't see clearly how to apply it to the simple case here. Any suggestions? – Baldrick Jul 5 '18 at 20:43
• Chi-square goodness-of-fit will answer the question of whether correct answers are distributed evenly among questions, but not whether a specific question is easier than another. If this is an acceptable approach, look it up. The chi-square GoF test is about as elegantly simple a test as exists in statistics. – Todd D Jul 5 '18 at 22:38
• @Todd: So I could use chi-square to determine if the distribution of answers is different to what you'd expect if they were all equally difficult. But this would say nothing about specific questions. Would questions with large residuals not be the ones that deviated the most from the expected? Can I use that somehow? – Baldrick Jul 6 '18 at 6:57
• To truly prove that "some questions are harder," you would also need to consider who answered the questions correctly/incorrectly. If there is one kid who answered 9/10 correctly, their relative intelligence is assumed to be higher than someone who answered 0/10 correctly. However, you are giving equal weight to the answering ability of both kids. This is actually an interesting optimization problem, often used to rank "relative toughness" of college courses (i.e. proving engineering courses really are harder than those silly humanities courses). – ERT Jul 7 '18 at 0:01

The data your provide, in isolation, can be viewed as a 1x10 table of cell probabilities or counts. As is, the data can be used to answer the question of whether all the questions are equally difficult. This can be done with a chi-square goodness-of-fit test. Using R:

scores <- c(60, 50, 40, 5, 90, 50, 35, 30, 80, 10)
exp <-  rep( c(sum(scores)/length(scores)), length(scores) )
chisq.test(scores)


Results in:

    Chi-squared test for given probabilities

data:  scores
X-squared = 148.89, df = 9, p-value < 2.2e-16


The p-value is far less than 0.01, which confirms visual inspection of the data.

• Thanks very much for your answer! I'll have to give it some thought and look at the other answer in more detail before I accept! – Baldrick Jul 8 '18 at 20:06

I think the way you want to approach this problem is with Cochran's Q test. This test can be seen as an extension of McNemar's test, extended to multiple instances (questions in this case), when the response is binary (correct / incorrect in this case). Pairwise McNemar tests may be an appropriate post-hoc test.

To see why this approach is better than a goodness-of-fit chi-square test, we can look at comparing two questions, and seeing that we could get the same frequencies whether or not one question is systematically more difficult than the other.

Here, I'll assume there are 100 students, with 60 getting Q1 correct and 50 getting Q2 correct.

Optional technical note: the zero in the first table may violate assumptions of the test, but I'm ignoring this for this example. An exact binomial test could be used in this case.

Let's consider a case in which Q2 really is more difficult than Q1. 60 students get Q1 correct and 50 students get Q2 correct. But more importantly, every student who got Q1 wrong got Q2 wrong, and 10 students who got Q1 correct got Q2 wrong. This is good evidence that the change in success from Q1 to Q2 all goes in one direction: Q2 is more difficult. McNemar's test confirms this with a low p-value. Cohen's g and odds ratio (neither shown) also indicate that the effect size is very large*.

### Q2 is more difficult than Q1
### Everyone who got Q1 wrong, also got Q2 wrong,
###   and another 10 people got Q2 wrong who got Q1 right.

Input =("
Question      Q2.correct   Q2.wrong
Q1.correct    50           10
Q1.wrong       0           40
")

row.names=1))

sum(Matrix.1)

### [1] 100

mcnemar.test(Matrix.1)

### McNemar's Chi-squared test with continuity correction
###
### McNemar's chi-squared = 8.1, df = 1, p-value = 0.004427


Now the case where neither question is more difficult. 60 students get Q1 correct and 50 students get Q2 correct. But more importantly, relatively many students who got Q1 correct got Q2 wrong, and relatively many students who got Q1 wrong got Q2 correct. This suggests that there is no systematic difference between the questions. McNemar's test agrees with a relatively high p-value. Cohen's g and odds ratio (neither shown) also indicate that the effect size is small*.

### Q2 is not more difficult than Q1
### Many who got Q1 wrong got Q2 correct, and vice versa

Input =("
Question      Q2.correct   Q2.wrong
Q1.correct    35           25
Q1.wrong      15           25
")

row.names=1))

sum(Matrix.2)

### [1] 100

mcnemar.test(Matrix.2)

### McNemar's Chi-squared test with continuity correction
###
### McNemar's chi-squared = 2.025, df = 1, p-value = 0.1547


.* For Cohen's g and odds ratio statistics,

library(rcompanion)

cohenG(Matrix.1)

cohenG(Matrix.2)


For interpretation, see this site.

• Thanks very much for your answer! I'll have to give it some thought and look at the other answer in more detail before I accept! – Baldrick Jul 8 '18 at 20:06
• I didn't provide any data about which students got what right and wrong - just the overall scores. Are you saying we'd need to include that data in to do this test? – Baldrick Jul 8 '18 at 20:09
• Yes, to use either McNemar or Cochran Q, you'd have to know the outcome for each student for each question. – Sal Mangiafico Jul 8 '18 at 22:08

The $\chi^2$ idea only tells you that coin tossing was not involved. It won't get you to what you want to know, which is the rank ordering of the questions in terms of difficulty.

The problem here is that you don't have that data or at least not as disclosed here. Imagine that the highest set of grades were 90's and nobody who got a 90 also got question 4 correct. Imagine the people with the lowest grades tended to get question 4 correct. You need to account for talent as well as percent who got the question correct. So you should have a $100\times{10}$ matrix, not a $1\times{10}$ matrix.

Furthermore, question 4, if it was multiple choice, was failed at a rate less than chance. The question may not be difficult; it may have been taught badly.

You could invert the idea in Item Response Theory, which assesses the ability of the person to instead assess the difficulty of the test items. You would use logistic regression on each item with overall grade by person as an independent variable.

The difficulty you will have is that it is unlikely that your questions are independent. If two questions ask about the same concept, then they might be the same question. It may be the case that if you miss question 2, then you are guaranteed to miss question 3.

Before you do try logistic regression, you will want to check for internal dependencies. Although inference does not seem to be your goal, you will want to do corrections on your statistical tests for multiple comparisons such as the Holm-Bonferroni.

If your questions are reasonably fair, with respect to course content, and have adequate independence, then you will likely just end up with the existing rank ordering. Because you have an endogeneity issue, since getting the question correct determines the grade and not the grade determines if the question is correct, you should bring in some external source of information on talent as an instrumental variable, something correlated with the grade in the class, but not with the specific set of questions.