How to calculate absolute fit indices (RMSEA, GFI...) from relative ones (AIC, BIC...)?

Question

I have conducted an IRT analysis with Conquest in order to compare two models (1-dimensional vs. 8-dimensional) applied to a given data set (41 items of a questionnaire, N=195).

Comparing the deviance scores (as well as resulting AIC/BIC scores), I know that the 8-dimensional model is superior in a relative sense.

How do I know if this model describes the data well enough in an absolute sense? My idea would be to calculate fit indices like GFI, NNFI etc. but I don't have a clue how to get them from my Conquest output.

Answer 1

Although I am not familiar with Conquest, as it is computing deviance scores and AIC/BIC scores it is likely computing a log-likelihood (LL) that you can use to compute what is usually called a "pseudo r-squared" statistic. For example, McFadden's rho-squared is given by:

rho^2 = 1 - LL_model / LL_null

where LL_model is the log-likelihood of your model and LL_null is the log-likelihood computed for a simple null model (e.g., one which only has intercepts).

If you cannot find the log-likelihood on the output you should be able to compute it backwards from the AIC (i.e., LL - AIC/-2 + k, where k is the number of parameters).

Also, keep in mind that although computing a pseudo r-squared will give you an absolute measure of fit there is, ultimately, no sensible criteria for determining whether it fits the data "well enough". Although there are some problems where a specific level of accuracy may be required, in most applications of statistics (and science) a good model is one that performs better than the other models, and the AIC and BIC are more appropriate for answering such questions than is an absolute measure of fit.