26
$\begingroup$

Context

I have been reading about item response theory, and I find it fascinating. I believe I understand the basics, but I am left wondering how to apply statistical techniques related to the area. Below are two articles that are similar to the area I would like to apply ITR in:

The second being the one I would actually like to extend at this point in time.

I have downloaded a free program called jMetrik, and it seems to be working great. I think it may be too basic as far as IRT goes, but I am unsure.

I know the "best" way would likely involve learning R; however, I don't know if I can spare the time to tackle that learning curve. Note that we have some funding to purchase software, but from what I see, there doesn't seem to be any great IRT programs out there.

Questions

  • What are your thoughts on the effectiveness of jMetrik?
  • How would you suggest I go forward in applying IRT?
  • What are the best programs for applying IRT?
  • Do any of you use IRT regularly? If so, how?
$\endgroup$
4
  • 1
    $\begingroup$ what software do you currently use? $\endgroup$
    – StasK
    Commented Sep 15, 2011 at 16:01
  • $\begingroup$ I have been using jMetrik. It is quite new it looks like and has been doing a lot of the things I am interested in! $\endgroup$
    – Behacad
    Commented Sep 15, 2011 at 23:36
  • 3
    $\begingroup$ Does it work in a script mode? If you only have a GUI, then your results will be very difficult to reproduce. A script mode is a must for any serious software. $\endgroup$
    – StasK
    Commented Sep 16, 2011 at 14:27
  • $\begingroup$ It has a log export feature. $\endgroup$
    – Will
    Commented Jun 2, 2020 at 14:02

6 Answers 6

26
$\begingroup$

As a good starter to IRT, I always recommend reading A visual guide to item response theory.

A survey of available software can be found on www.rasch.org.

From my experience, I found the Raschtest (and associated) Stata command(s) very handy in most cases where one is interested in fitting one-parameter model. For more complex design, one can resort on GLLAMM; there's a nice working example based on De Boeck and Wilson's book, Explanatory Item and Response Models (Springer, 2004).

About R specifically, there are plenty of packages that have become available in the past five years, see for instance the related CRAN Task View. Most of them are discussed in a special issue of the Journal of Statistical Software (vol. 20, 2007). As discussed in another response, the ltm and eRm allow to fit a wide range of IRT models. As they rely on different method of estimation---ltm used the marginal approach while eRm use the conditional approach---choosing one or the other is mainly a matter of the model you want to fit (eRm won't fit 2- or 3-parameter models) and the measurement objective you follow: conditional estimation of person parameters has some nice psychometric properties while a marginal approach let you easily switch to mixed-effects model, as discussed in the following two papers:

There are also some possibilities to fit Rasch models using MCMC methods, see e.g. the MCMCpack package (or WinBUGS/JAGS, but see BUGS Code for Item Response Theory, JSS (2010) 36).

I have no experience with SAS for IRT modeling, so I'll let that to someone who is more versed into SAS programming.

Other dedicated software (mostly used in educational assessment) include: RUMM, Conquest, Winsteps, BILOG/MULTILOG, Mplus (not citing the list already available on wikipedia). None are free to use, but time-limited demonstration version are proposed for some of them. I found jMetrik very limited when I tried it (one year ago), and all functionalities are already available in R. Likewise, ConstructMap can be safely replaced by lme4, as illustrated in the handout linked above. I should also mention mdltm (Multidimensional Discrete Latent Trait Models) for mixture Rasch models, by von Davier and coll., which is supposed to accompagny the book Multivariate and Mixture Distribution Rasch Models (Springer, 2007).

$\endgroup$
6
  • $\begingroup$ Fantastic! Thank you for the very comprehensive answer! Both of these answers will help me nicely, as well as anyone else who is curious in pursuing this area I'm sure. I do suggest you take a look at jMetrik again, and perhaps give me your thoughts. It has Rasch modelling now, ICC curves, some IRT curves, IRT equating (1PT, 2PT, 3PT models), and more. $\endgroup$
    – Behacad
    Commented Sep 19, 2011 at 15:30
  • $\begingroup$ I will look at the updated version. But, honestly, what @Stask said is worth to remember if you plan to do anything serious beyond parameters estimation and visual reporting. To get an idea of what I mean, look at An introduction to psychometric theory with applications in R. (It covers more than just what some call "modern" psychometrics.) $\endgroup$
    – chl
    Commented Sep 19, 2011 at 18:05
  • $\begingroup$ @chi - I got very different results when I ran IRT analyses in R using ltm compared to when I ran the same analysis in Mplus and then in Multilog (whereas analysis in Mplus and Multilog were identical). I tried to work out why with the author of ltm, but did not receive any responses. Have you had any similar experiences with discrepant results from different software packages? $\endgroup$
    – Tormod
    Commented Sep 20, 2011 at 9:12
  • $\begingroup$ @Tormod Very different in what sense? Running the ex5.5 example in Mplus and R/ltm, I got approx. the same results. Some variations depending on software being used are to be expected, though. Usually, what I've found is related to the choice of the parametrization (logit/probit), whether it's $\theta_p-\beta_i$ ($i=1,\dots,k$ items, $p=1,\dots,n$ persons) or $\beta_i-\theta_p$ that is modeled, and the estimation method (marginal vs. conditional likelihood), and for 2+ parameter models whether thresholds are centered or not. $\endgroup$
    – chl
    Commented Sep 20, 2011 at 11:33
  • $\begingroup$ @chi - sorry for the delay in response, I haven't noticed your comment. Don't know if this is still relevant, but I got differences varying from 0.184 - 1.429 for the discrimination parameters (a). For example, Mplus gave an a=5.084 whereas ltm gave the same item an a=3.655. Overall, ltm gave smaller a's than mplus. (Running the same analyses in multilog gave a's corresponding to the a's from Mplus). The b's were more similiar. $\endgroup$
    – Tormod
    Commented Oct 28, 2011 at 12:58
8
$\begingroup$

To the first question, I don't have any information about jMetrick.

In applying IRT, (as with any other statistical procedure) the first step is to use it with as many different kinds of data as possible. There is a learning curve, but I believe that it is worth it.

One important feature of IRT is the differentiation between Rasch models and IRT models. They were developed by different people for different purposes. That being said, IRT models are a superset of Rasch models.

Rasch models are one parameter models - they assume that all items on a questionnaire are equally predictive of the latent trait.

IRT models, however are two parameter models which allow the questions to differ in their ability to provide information about the ability of participants.

In addition, there are three parameter models which are like the IRT models, except that they allow for a guessing parameter to account for participants ability to get the right answer by chance (this is more of a concern in ability rather than personality tests).

In addition, there is multidimensional IRT which estimates multiple latent abilities at once. I don't know much about this, but its an area which I intend to learn more.

There is also a distinction between dichotomous and polytomous IRT methods. Dichotomous IRT models are those used in ability tests, which have a right and wrong answer. Polytomous IRT models are used in personality tests, where there are multiple answers, which are equally right (in the sense that there is no correct answer).

I personally use R for item response theory. There are two main packages that I have used, eRm which fits Rasch models only, and ltm which fits item response theory models (two and three parameter models). Both have similiar functionality, and both provide more routines for dichotomous IRT models. I don't know if R is the "best" for IRT, it does not have all of the multitude of IRT models available, but it is certainly the most extensible, in that one can program these models relatively easily.

I use IRT almost exclusively for polytomous models, in R. I typically start with non parametric IRT methods (provided in the package mokken) to test the assumptions, and then proceed with a rasch model, adding more complexity as required to get good fit.

For multidimensional IRT, there is the package `mirt', which provides this functionality. I have not used it so I cannot really comment.

If you do install these packages into R, and call the 'vignette("packagename")' function then you should get some useful vignettes (definitely for eRm and mokken, possibly for the others) which may prove useful for you (depending on your level of mathematical sophistication).

Finally, there are a number of good books available for rasch and irt models. Item response theory for psychologists is often used (though I didn't like the style), and further up the technical sophistication chain, there are two extremely comprehensive and useful textbooks - the Handbook of Modern Item Response Theory and Rasch Models:Foundations, Recent Developments and Applications.

I hope this helps.

$\endgroup$
1
  • $\begingroup$ Thank you! That is much appreciated. I would also like to hear more about software options if someone has some knowledge in the area. $\endgroup$
    – Behacad
    Commented Sep 15, 2011 at 18:54
3
$\begingroup$

jMetrik is more powerful than you may think. It is designed for operational work where researchers need multiple procedures in a single unified framework. Currently you can estimate IRT parameters for the Rasch, partial credit and rating scale models. It also allows for IRT scale linking via the Stocking-Lord, Haebara and other methods. Because it includes an integrated database, the output from the IRT estimation can be used in scale linking without the need to reshape data files. Moreover, all output can be stored in the database for use with other methods in jMetrik or external programs like R.

You can also run it with scripts instead of the GUI. For example, the follwing code will (a) import data into the database, (b) score items with an answer key, (c) estimate Rasch model parameters, and (d) export data as a CSV file. You can use the final output file as input into R for further analysis, or you can use R to connect directly to the jMetrik database and work with the results.

#import data into database
import{
     delimiter(comma);
     header(included);
     options(display);
     description();
     file(C:/exam1-raw-data.txt);
     data(db = testdb1, table = EXAM1);
}

#conduct item scoring with the answer key
scoring{
     data(db = mydb, table = exam1);
     keys(4);
     key1(options=(A,B,C,D), scores=(1,0,0,0), variables=  (item1,item9,item12,item15,item19,item21,item22,item28,item29,item30,item34,item38,item42,item52,item55));
     key2(options=(A,B,C,D), scores=(0,1,0,0), variables=(item4,item6,item16,item18,item24,item26,item32,item33,item35,item43,item44,item47,item50,item54));
     key3(options=(A,B,C,D), scores=(0,0,1,0), variables=(item3,item5,item7,item11,item14,item20,item23,item25,item31,item40,item45,item48,item49,item53));
     key4(options=(A,B,C,D), scores=(0,0,0,1), variables=(item2,item8,item10,item13,item17,item27,item36,item37,item39,item41,item46,item51,item56));
}

#Run a Rasch models analysis.
#Item parameters saved as database table named exam1_rasch_output
#Residuals saved as a databse table named exam1_rasch_resid
#Person estimates saved to original data table. Person estimate in variable called "theta"
rasch{
     center(items);
     missing(ignore);
     person(rsave, pfit, psave);
     item(isave);
     adjust(0.3);
     itemout(EXAM1_RASCH_OUTPUT);
     residout(EXAM1_RASCH_RESID);
     variables(item1, item2, item3, item4, item5, item6, item7, item8, item9, item10, item11, item12, item13, item14, item15, item16, item17, item18, item19, item20, item21, item22, item23, item24, item25, item26, item27, item28, item29, item30, item31, item32, item33, item34, item35, item36, item37, item38, item39, item40, item41, item42, item43, item44, item45, item46, item47, item48, item49, item50, item51, item52, item53, item54, item55, item56);
     transform(scale = 1.0, precision = 4, intercept = 0.0);
     gupdate(maxiter = 150, converge = 0.005);
     data(db = testdb1, table = EXAM1);
}

#Export output table for use in another program like R
export{
     delimiter(comma);
     header(included);
     options();
     file(C:/EXAM1_RASCH_OUTPUT.txt);
     data(db = testdb1, table = EXAM1_RASCH_OUTPUT);
}

The software is still in its early stages of development. I am currently adding exploratory factor analysis and more advanced item response models. Unlike many other IRT programs, jMetrik is open source. all of the measurement procedures use the psychometrics library which is currently available on GitHub, https://github.com/meyerjp3/psychometrics. Anyone interested in contributing is welcomed.

$\endgroup$
1
$\begingroup$

Many important developments have been made in item response theory (IRT) software since you first asked your question, so an updated answer may be in order. This being said, a visual guide to item response theory remains a great reference, so I second @chl's recommendation$^1$.

Software packages mentioned above can be categorized as either IRT specialized software (e.g., BILOG/MULTILOG, Xcalibre, jMetrik, and R packages such as ltm) or more general purpose statistical software (e.g., MPLUS, SAS, STATA, and R packages such as lme4). While these packages differ in important ways, most can accommodate the most commonly used IRT models such as the two-parameter logistic (2PL) model, graded response model (GRM), Rasch/one-parameter logistic (1PL) model, & generalized partial credit model (GPCM)$^2$.

While opinions may vary, the state of the art IRT software packages today are flexMIRT - a proprietary software package that specializes in IRT models and related methods, and mirt an R package. Both packages not only can accommodate the IRT models mentioned above, they are also the most up to date when it comes to state of the art estimation algorithms and model fit information. For example, both implement the Metropolis-Hastings Robbins-Monro (MHRM) algorithm (Cai 2010a, & Cai 2010b), which allows for efficient estimation of high dimensional IRT models. Additionally, both incorporate recent developments in global model fit (e.g., see Cai, Chung, & Lee, 2023; and Maydeu-Olivares 2014 for more information) and piecewise model fit. For these reasons, flexMIRT and mirt are the IRT packages I use the most.

$^1$ A number of good books have also recently been published. Bonifay (2019) is a good introduction for readers new to IRT, and Bock & Gibbons (2021) is a good reference for individuals with more experience with psychological measurement and/or statistics.

$^2$ See Thissen & Steinberg (1986) for a great overview of IRT models.

References

Bock, R. D., & Gibbons, R. D. (2021). Item response theory. John Wiley & Sons.

Bonifay, W. (2019). Multidimensional item response theory. Sage Publications.

Cai, L. (2010). High-dimensional exploratory item factor analysis by a Metropolis–Hastings Robbins–Monro algorithm. Psychometrika, 75, 33-57.

Cai, L. (2010). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307-335.

Cai, L., Chung, S. W., & Lee, T. (2023). Incremental model fit assessment in the case of categorical data: Tucker–lewis index for item response theory modeling. Prevention Science, 24(3), 455-466.

Maydeu-Olivares, A. (2014). Evaluating the fit of IRT models. In Handbook of item response theory modeling (pp. 111-127). Routledge.

Thissen, D., & Steinberg, L. (1986). A taxonomy of item response models. Psychometrika, 51(4), 567-577.

$\endgroup$
0
$\begingroup$

You have quite a broad list of questions here, but quite relevant for many researchers!

I highly recommend you go forward in IRT, but only if your situation meets the requirements. For example, it fits well with the types of tests you use, and probably most importantly that you have the necessary sample sizes. For dichotomous multiple-choice data, I recommend the 3PL model (the Rasch argument of "objective measurement" is strikingly uncompelling), and 500-1000 is generally the minimum sample size. Dichotomous data without guessing, like psychological surveys that have Y/N responses to statements, work well with the 2PL. If you have rating scale or partial credit data, there are polytomous models designed specifically for those situations.

IMHO, the best program for applying IRT is Xcalibre. It is relatively user-friendly (simple GUI as well as some command-line batch-type if you want it for some reason) and produces highly readable output (MS Word reports with extensive tables and figures). I recommend against using R for the opposite reasons. The drawback, of course, is that it is not free, but you tend to get what you pay for as they say. Full description, example output, and a free trial are available at www.assess.com.

$\endgroup$
1
  • $\begingroup$ Would you like to elaborate on why the argument on "objective measurement" sensu Rasch (or more correctly the possibility of specific objective comparisons) is "strikingly uncompelling"? $\endgroup$
    – Momo
    Commented Apr 14, 2013 at 20:45
0
$\begingroup$

In the mean time there has published a new book by Frank Baker, Baker Frank B. , Seock-Ho Kim. The Basics of Item Response Theory Using R. Springer International Publishing (2017). It does not make use of R packages but offers snippets.

A (crowded) list of R packages for IRT with succinct description is available on CRAN.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.