4
$\begingroup$

I have a dataset where the difficulty level of my items and skill of my participants is unknown, but I would like to be able to extract some metric of how individual participants do on the test. What comes to mind initially is to use an item response theory model. However, I have two a priori constraints that makes me question whether IRT will be able to adapt to my peculiar case, or whether there is a better approach to use. As there are two constraints, I will split them into two questions. If these questions are too simple, please refer me to an entry level IRT text which will provide the answers. The first question is this:

I know that the difficulty of my items increases monotonically, by which I mean I know item A is harder than item B, and so on, but I do not know the extent to which item A is harder than item B. Is there a way to construct an item response theory model to adjust the model so that the item difficulty parameters are forced to reflect this?

P.S. I was unable to think of appropriate tags for this question, if you have some suggestions please either make the edits yourself or suggest them in comments.

$\endgroup$
5
$\begingroup$

Off the shelf, the various software offerings from SSI may do what you want.

If I wanted to enforce the constraint, and I didn't have too much data, then I'd switch into a BUGS / JAGS MCMC framework for modelling. I'd use the LSAT example in book 2 of the worked examples (contained in the JAGS distribution or as a separate download from the BUGS people) as the code for standard '3 parameter' IRT model. Then I'd add a line imposing the parameter constraint as described on p.35 of the current JAGS manual. I'd then curl up with a good book, since convergence on such models tends to take ages.

Or I might not bother. Standard errors for such parameters tend to be pretty tiny for a respectable number of subjects, so the ordering constraint may not make any difference to your estimation certainty. And if your parameters end up sitting on top of one another, they're probably trying to tell you your prior is wrong.

$\endgroup$
  • $\begingroup$ I don't think I have the funding to purchase anything from SSI. Just in case I find $250 under the couch, did you have a particular product in mind? $\endgroup$ – russellpierce Dec 10 '10 at 17:59
  • $\begingroup$ Thanks for the pointers. I guess JAGS MCMC it is; thankfully I can be very confident in the priors so hopefully something shakes out. I'm a little concerned because though I have lots of observations on the items I have relatively few subjects. I'll start throwing models at it and see how it shapes out. $\endgroup$ – russellpierce Dec 10 '10 at 18:00
  • $\begingroup$ (+1) A recent tutorial with BUGS/R can be found here: The Use of R and WinBUGS in Fitting Item Response Theory Models. $\endgroup$ – chl Dec 11 '10 at 14:45
  • $\begingroup$ Also see S. McKay Curtis, BUGS Code for Item Response Theory, Journal of Statistical Software August 2010, Volume 36, Code Snippet 1. $\endgroup$ – Jack Tanner Jan 18 '12 at 7:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.