# How to judge which test is more difficult?

I used computer to automatically select 30 questions from a database, and forward to 20 students. Then I reviewed their answers, and used 1 to represent a correct answer and 0 for an incorrect one. For each student, I got a 1-0 sequence with sequence length 30 (for instance, 11100011111111011110111101111). Here I denote the 20 1-0 sequences as Seq1, Seq2, ...,Seq20.

After that, I used computer to generate another 30 questions and perform the test again. I got another 20 1-0 sequences, denoted as Seq21, Seq22, ..., Seq40.

I would like to know how to judge whether the difficulty of the first test and the second test is significantly different based on Seq1, ..., Seq40 (the outcomes of the two tests). What statistical tool or hypothesis test shall I use to solve this problem?

• What do you mean by significant difficulty? You need to define that to be able to design a test. Also, your question title does not match your actual question. – Richard Hardy Mar 18 '15 at 10:42
• thanks,Richard,i mistype the word, it should be significantly difficult, i wanna know which test is more difficult for the students to answer based on the correctness of their answers, is binomial test applicable for this situation? – ericjp Mar 18 '15 at 10:51
• A binomial test is not suitable unless the individual questions within a test are equally difficult. – Glen_b -Reinstate Monica Mar 18 '15 at 12:25

$$P(X_{ij} = 1) = \frac{\exp(\theta_i - \beta_j)}{1-\exp(\theta_i - \beta_j)}$$
where individual response $Y_{ij}$ are modeled as a function of student ability $\theta_i$ and the item difficulty $\beta_j$. So you get estimates of both abilities and difficulties. The test with more difficult items is the more difficult. In R there are several packages for estimating IRT models, you can check ltm (Rizopoulos, 2006) and mirt (Chalmers, 2012) packages and their documentation for further information. IRT models are estimable as well in other software e.g. MPLUS, MIRT (by Cees A.W. Glas) etc.