I have conducted an IRT analysis with Conquest in order to compare two models (1-dimensional vs. 8-dimensional) applied to a given data set (41 items of a questionnaire, N=195).

Comparing the deviance scores (as well as resulting AIC/BIC scores), I know that the 8-dimensional model is superior in a relative sense.

How do I know if this model describes the data well enough in an absolute sense? My idea would be to calculate fit indices like GFI, NNFI etc. but I don't have a clue how to get them from my Conquest output.

  • 1
    $\begingroup$ Could you confirm that "1- vs. 8-dimensional" means we are dealing with uni- vs. multidimensional models? $\endgroup$ – chl Nov 25 '12 at 21:04

Although I am not familiar with Conquest, as it is computing deviance scores and AIC/BIC scores it is likely computing a log-likelihood (LL) that you can use to compute what is usually called a "pseudo r-squared" statistic. For example, McFadden's rho-squared is given by:

rho^2 = 1 - LL_model / LL_null

where LL_model is the log-likelihood of your model and LL_null is the log-likelihood computed for a simple null model (e.g., one which only has intercepts).

If you cannot find the log-likelihood on the output you should be able to compute it backwards from the AIC (i.e., LL - AIC/-2 + k, where k is the number of parameters).

Also, keep in mind that although computing a pseudo r-squared will give you an absolute measure of fit there is, ultimately, no sensible criteria for determining whether it fits the data "well enough". Although there are some problems where a specific level of accuracy may be required, in most applications of statistics (and science) a good model is one that performs better than the other models, and the AIC and BIC are more appropriate for answering such questions than is an absolute measure of fit.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.