Assess overall question difficulty in Graded Response Model

Question

I used graded response modelling (GRM) to assess the difficulty of items from a survey using a Likert scale from 1 to 5.

The results show the difficulty of each item (i.e., 1-5), for each question.

I am wondering if there is any method to combine these scores into one difficulty score, for each question.

For example, by averaging the individual results, or any other method?

Thank you!

Answer 1

Ali, Chang, and Anderson (2015) provided a few such generalized difficulty measures (one of which is close to what you suggested by taking the average of the estimated difficulty parameters), though I'll only describe the one that I believe is most optimal in terms of its interpretation.

Difficulty in unidimensional dichotomous items is generally understood as the location where a given latent trait $\theta$ would result in an expected probability of $P(y=1|\theta)=1/2$. This means, for instance, that more difficult items require participants to have high-ability in order to obtain a 50% chance of a question correctly; the converse interpretation is true for less difficult items, in which case only low abilities are required. Classical parameterizations of IRT models often contain some variant of a difficulty parameter which satisfies this expression immediately.

The same interpretation holds when searching for the conditional expected value of the response function, $E(y|\theta)$, where the same information is carried through since in this case $P(y=1|\theta) = E(y|\theta)$.

Generalizing this idea with respect to finding a suitable $\theta$ location w.r.t. the conditional expectation for polytomous items with $C$ categories, Ali, Chang, and Anderson (2015) proposed locating the $\theta$ that would correspond to the solution $$E(y|\theta)= \frac{(C-1)}{2}$$ This generally requires some numerical search to locate, which was presented in the authors' article. Note that dichotomous responses ($C=2$) are just special cases of this expression, where the goal would be to find some $\theta$ that solves $E(y|\theta)= 1/2$.

Below is some code from the mirt package's implementation of this equation's solution for several graded response models, where each item had $C=4$ categories.

> library(mirt)
> mod <- mirt(Science, 1)
Iteration: 36, Log-Lik: -1608.870, Max-Change: 0.00010

> # discrimination-difficulty parameterization
> coef(mod, simplify=TRUE, IRTpars = TRUE)$items
               a        b1         b2        b3
Comfort 1.041755 -4.669193 -2.5341299 1.4072541
Work    1.225962 -2.385068 -0.7350678 1.8488053
Future  2.293372 -2.282226 -0.9652918 0.8562529
Benefit 1.094915 -3.057698 -0.9056673 1.5419094

> gen.difficulty(mod)
   Comfort       Work     Future    Benefit 
-2.3089094 -0.5741303 -0.9207845 -0.8530161 

> colMeans(Science)
 Comfort     Work   Future  Benefit 
3.119898 2.721939 2.989796 2.836735 

So in this case the Work item appears to be the most difficult on average, while the Comfort item was the easiest, which incidentally corresponds to the mean of the raw response observations (not required, as these generalized difficulty measures are more general, but for ordinal data this is expected).

References

Ali, U. S., Chang, H.-H., & Anderson, C. J. (2015). Location indices for ordinal polytomous items based on item response theory (ResearchReport No. RR-15-20). Princeton, NJ: Educational Testing Service. http://dx.doi.org/10.1002/ets2.12065