Item Response Models with Few Measures

Question

Are there any problems with using Item Response Theory models such as the Graded Response Model when the number of measures is small? I don't know if there are any asymptotic results that would make a Graded Response Model undesirable with such a small number of measures.

Answer 1

There are a number of issues, include convergence failures, biased parameter estimates, large standard errors, and difficulty in using the fitted model for predictive purposes (i.e., for scoring individuals by obtaining $\hat{\theta}$ estimates).

These are all technical issues, though more philosophically you can think of other issues such as failure to sufficiently sample the domain, validity issues in selecting only certain kinds of response stimuli types, and so on.

So yes, there are potentially many issues, though these depend of philosophical considerations as well as data characteristics.

Answer 2

I would like to complement the above answer, by noting that estimation problems in 2PL Graded Response Model (GRM) may also depend on the issues of sample size and item response format. Note: there is no golden rule for 2PL GRM sample size, but some simulation studies (Reise & Yu, 1990) recommend that with a 5-category Likert scale, at least 500 respondents are needed to produce accurate parameter estimates in terms of bias. This is also briefly discussed by (Thorpe & Favia 2012, p.11)

Also, violation of the IRT assumptions can lead to serious problems in parameter estimation. First, the essential unidimensionality assumption, according to which items are measured by a single latent factor, is crucial in IRT modelling (Embretson & Reise, 2000), and its violation will negatively affect accuracy of estimated parameters. If your data is not essentially unidimensional you can try fitting multidimentional IRT models as an option. Second, there is an assumption of Local Independence, i.e.Items are unrelated after controlling for factors. There is substantial empirical body of evidence suggesting that statistical analysis with locally dependent items is misleading (Chen & Wang, 2007; Chen & Thissen, 1997; Junker, 1991; Sireci, Thissen, Steinberg, & Mooney, 1989; Tuerlinckx & De Boek, 1998; Yen, 1993).