Comparison indices of CFA

I am comparing different models in Confirmatory Factor Analysis (CFA) to decide what my optimal number of factors and factor structure should be. The main indices I have been using are chi square , Comparative Fit Index (CFI) , RMSEA and Incremental Fit index (IFI. I am aware that there is cut off values available for these indices. However, I am unsure if there is any way of saying statistically if one of the models is better than the other.

Preamble

As @Paul Christiansen correctly points out, when trying to select among several confirmatory factor analysis (CFA) models, it is very important to select a model based not only on statistical tests and/or fit indices but also on theoretical considerations. This is not only because theoretical considerations are important but also because some models with very different theoretical interpretations have a tendency to fit the data better than others (e.g., see Bonifay & Cai, 2017. Their article does not directly concern CFA, though their findings can be generalized to a CFA context).

All this being said, the fit indices you cite, the $$\chi^2$$ difference test, Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI), and Incremental Fit Index (IFI)$$^1$$ can be used to compare competing models. However, it should be noted that not all are necessarily formal tests for comparing competing models. The $$\chi^2$$ difference test is$$^2$$. However, the CFI and RMSEA are not necessary. Typically, one just compares competing models' CFI and/or RMSEA values and chooses the model with the better value (see Lai, 2020 for information on confidence intervals for RMSEA and CFI differences among competing models can be computed).

$$^1$$ Note that the IFI is not a fit index per se. Instead, it is a family of fit indices that includes the CFI.

$$^2$$ See Pavlov et al. (2020) for best practices.

References

Bonifay, W., & Cai, L. (2017). On the complexity of item response theory models. Multivariate behavioral research, 52(4), 465-484.

Lai, K. (2020). Confidence interval for RMSEA or CFI difference between nonnested models. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 16-32.

Pavlov, G., Shi, D., & Maydeu-Olivares, A. (2020). Chi-square difference tests for comparing nested models: An evaluation with non-normal data. Structural Equation Modeling: A Multidisciplinary Journal, 27(6), 908-917.

AIC values can be used to compare the models, smaller values suggesting a better model. AIC values are a function of the log-likelihood and therefore can only be compared if the models are fitted on the same data set (which I guess you have done). There's no agreed cut point to what is a substantial AIC difference, I have seen at least 2, at least 4 and at least 10 used.

Often people just compare the fit and select the better fitting values although that can be problematic inasmuch as sometimes fit indices can contradict eg CFI being better in model A, RMSEA being better in model B.

Also, you may have very small differences which may suggest a more complicated and perhaps less practically useful structure is better, so you can be guided by theoretical and practical rationales.

Anderson, D., & Burnham, K. (2004). Model selection and multi-model inference. Second. NY: Springer-Verlag, 63(2020), 10.

general AIC info https://en.wikipedia.org/wiki/Akaike_information_criterion

• Hi, welcome to CV ! Could you please provide a reference for your first link so that people can still access the information if the link dies in the future ? Thx in advance ! Commented Aug 24, 2022 at 8:23