Starting with randomly generated person ability parameters $\textbf{z}$ and item parameters $(\boldsymbol \alpha,\boldsymbol{\beta})$, I simulated data representing correct/incorrect responses of $N$ people on $M$ items.

Using the $\text{mirt}$ R package, I can derive estimates $\boldsymbol{z^*}, \boldsymbol{\alpha^*},\boldsymbol{\beta^*}$ of the aforementioned parameters, which effectively constitute a person/item model.

Though there are plenty of fit statistics that characterize the extent to which the model fits the data, I'm more interested in the fidelity of estimated parameters to the parameters used to generate the simulated data.

That is, how close (in some sense) is $\boldsymbol \alpha$ to $\boldsymbol{\alpha^*}$, for instance.

I'm wondering if this can this be a good measure of model fitness, at least in the case of simulated data.


It is not uncommon in IRT to use simulation studies to measure model fitness. You would call $z$, $(\alpha,\beta)$ your true values, and surely you would like to find out how well your model performs in uncovering these. Note that you would do that with multiple simulated data, obtaining many instances of estimates $z^*_i, \alpha^*_i, \beta^*_i$. You can compare those to the true values for example using the RMSE (root mean square error).

For a nice practical example, see Kang, Cohen and Sung (2009). For discussions about simulation studies in IRT, have a look at Luecht and Ackerman (2018) and Yen (1981).

Kang, T., Cohen, A. S., & Sung, H.-J. (2009). Model Selection Indices for Polytomous Items Applied Psychological Measurement, 33, 499-518.

Luecht, R., & Ackerman, T. A. (2018). A technical note on IRT simulation studies: Dealing with truth, estimates, observed data, and residuals. Educational Measurement: Issues and Practice.

Yen, W. M. (1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5, 245–262.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.