How to get started with applying item response theory and what software to use?

Question

Context

I have been reading about item response theory, and I find it fascinating. I believe I understand the basics, but I am left wondering how to apply statistical techniques related to the area. Below are two articles that are similar to the area I would like to apply ITR in:

• http://www.jstor.org/stable/4640738?seq=7
• http://www.ncbi.nlm.nih.gov/pubmed/21744971

The second being the one I would actually like to extend at this point in time.

I have downloaded a free program called jMetrik, and it seems to be working great. I think it may be too basic as far as IRT goes, but I am unsure.

I know the "best" way would likely involve learning R; however, I don't know if I can spare the time to tackle that learning curve. Note that we have some funding to purchase software, but from what I see, there doesn't seem to be any great IRT programs out there.

Questions

• What are your thoughts on the effectiveness of jMetrik?
• How would you suggest I go forward in applying IRT?
• What are the best programs for applying IRT?
• Do any of you use IRT regularly? If so, how?

Answer 1

As a good starter to IRT, I always recommend reading A visual guide to item response theory.

The FreeIRT project features a list of dedicated macros or packages to be used with SAS, Stata, R, among others. Another survey of available software can be found on www.rasch.org.

From my experience, I found the Raschtest (and associated) Stata command(s) very handy in most cases where one is interested in fitting one-parameter model. For more complex design, one can resort on GLLAMM; there's a nice working example based on De Boeck and Wilson's book, Explanatory Item and Response Models (Springer, 2004).

About R specifically, there are plenty of packages that have become available in the past five years, see for instance the related CRAN Task View. Most of them are discussed in a special issue of the Journal of Statistical Software (vol. 20, 2007). As discussed in another response, the ltm and eRm allow to fit a wide range of IRT models. As they rely on different method of estimation---ltm used the marginal approach while eRm use the conditional approach---choosing one or the other is mainly a matter of the model you want to fit (eRm won't fit 2- or 3-parameter models) and the measurement objective you follow: conditional estimation of person parameters has some nice psychometric properties while a marginal approach let you easily switch to mixed-effects model, as discussed in the following two papers:

• Doran, H., Bates, D., Bliese, P. and Dowling, M. (2007). Estimating the Multilevel Rasch Model: With the lme4 Package. Journal of Statistical Software, 20(2). See also Doug Bates's slides on R-forge
• De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuerlinckx, F., and Partchev, I. (2011). The Estimation of Item Response Models with the lmer Function from the lme4 Package in R. Journal of Statistical Software, 39(12). See also the aforementioned De Boeck's handbook and this handout

There are also some possibilities to fit Rasch models using MCMC methods, see e.g. the MCMCpack package (or WinBUGS/JAGS, but see BUGS Code for Item Response Theory, JSS (2010) 36).

I have no experience with SAS for IRT modeling, so I'll let that to someone who is more versed into SAS programming.

Other dedicated software (mostly used in educational assessment) include: RUMM, Conquest, Winsteps, BILOG/MULTILOG, Mplus (not citing the list already available on wikipedia). None are free to use, but time-limited demonstration version are proposed for some of them. I found jMetrik very limited when I tried it (one year ago), and all functionalities are already available in R. Likewise, ConstructMap can be safely replaced by lme4, as illustrated in the handout linked above. I should also mention mdltm (Multidimensional Discrete Latent Trait Models) for mixture Rasch models, by von Davier and coll., which is supposed to accompagny the book Multivariate and Mixture Distribution Rasch Models (Springer, 2007).

Answer 2

To the first question, I don't have any information about jMetrick.

In applying IRT, (as with any other statistical procedure) the first step is to use it with as many different kinds of data as possible. There is a learning curve, but I believe that it is worth it.

One important feature of IRT is the differentiation between Rasch models and IRT models. They were developed by different people for different purposes. That being said, IRT models are a superset of Rasch models.

Rasch models are one parameter models - they assume that all items on a questionnaire are equally predictive of the latent trait.

IRT models, however are two parameter models which allow the questions to differ in their ability to provide information about the ability of participants.

In addition, there are three parameter models which are like the IRT models, except that they allow for a guessing parameter to account for participants ability to get the right answer by chance (this is more of a concern in ability rather than personality tests).

In addition, there is multidimensional IRT which estimates multiple latent abilities at once. I don't know much about this, but its an area which I intend to learn more.

There is also a distinction between dichotomous and polytomous IRT methods. Dichotomous IRT models are those used in ability tests, which have a right and wrong answer. Polytomous IRT models are used in personality tests, where there are multiple answers, which are equally right (in the sense that there is no correct answer).

I personally use R for item response theory. There are two main packages that I have used, eRm which fits Rasch models only, and ltm which fits item response theory models (two and three parameter models). Both have similiar functionality, and both provide more routines for dichotomous IRT models. I don't know if R is the "best" for IRT, it does not have all of the multitude of IRT models available, but it is certainly the most extensible, in that one can program these models relatively easily.

I use IRT almost exclusively for polytomous models, in R. I typically start with non parametric IRT methods (provided in the package mokken) to test the assumptions, and then proceed with a rasch model, adding more complexity as required to get good fit.

For multidimensional IRT, there is the package `mirt', which provides this functionality. I have not used it so I cannot really comment.

If you do install these packages into R, and call the 'vignette("packagename")' function then you should get some useful vignettes (definitely for eRm and mokken, possibly for the others) which may prove useful for you (depending on your level of mathematical sophistication).

Finally, there are a number of good books available for rasch and irt models. Item response theory for psychologists is often used (though I didn't like the style), and further up the technical sophistication chain, there are two extremely comprehensive and useful textbooks - the Handbook of Modern Item Response Theory and Rasch Models:Foundations, Recent Developments and Applications.

I hope this helps.

Answer 3

You have quite a broad list of questions here, but quite relevant for many researchers!

I highly recommend you go forward in IRT, but only if your situation meets the requirements. For example, it fits well with the types of tests you use, and probably most importantly that you have the necessary sample sizes. For dichotomous multiple-choice data, I recommend the 3PL model (the Rasch argument of "objective measurement" is strikingly uncompelling), and 500-1000 is generally the minimum sample size. Dichotomous data without guessing, like psychological surveys that have Y/N responses to statements, work well with the 2PL. If you have rating scale or partial credit data, there are polytomous models designed specifically for those situations.

IMHO, the best program for applying IRT is Xcalibre. It is relatively user-friendly (simple GUI as well as some command-line batch-type if you want it for some reason) and produces highly readable output (MS Word reports with extensive tables and figures). I recommend against using R for the opposite reasons. The drawback, of course, is that it is not free, but you tend to get what you pay for as they say. Full description, example output, and a free trial are available at www.assess.com.