SEM Goodness of Fit and model refinement

Question

I am currently working on a theory building mixed-methods paper in which I estimate a structural equation path model based on suggestive paths and links drawn from qualitative in-depth interviews.Based on existing literature and the insights from my qualitative analyses, I have built a structural equation model that is composed of the following:

1. One binary predictor variable (non-latent)

2. 1 latent outcome variable with three items (ordinal scale)

3. 1 latent variable with seven items (ordinal scale)

4. 1 latent variable with two items(continuous scale)

5. 1 latent variable with 14 items (ordinal scale)

6. 1 latent variable with 10 items (ordinal scale)

The sample size is 522 and for each of the latent variables, individual items are highly significant. However, goodness of fit for the full SEM (as well as for some of the individual measurement models for latent variables) is not optimal. According to CFI , model fit is very good (CFI=0.986), but according to RMSEA (=0.103) and SRMR (=0.104) it is rather poor, also chi2 is 323404.72*** with df=754 indicates poor fit.

I have already inspected modification indices and correlated some error terms, but the model fit could not be improved substantially. The model is very much embedded in theory and corresponds to my qualitative findings and I am therefore hesitant to discard it altogether. I was therefore wondering whether you could give me some advice or share your thoughts on the following questions:

**1) Do you think the above fit statistics are 'too bad' to include the model in a paper/send it to a journal?

2) Is there any reason for why model fit looks good according to CFI but poor according to RMSEA and SRMR?

3) Do you have any other advice on how to possibly refine the model without neglecting theoretical foundations?**

Answer 1

A few suggestions/clarifications before directly addressing your questions.

First, a significant $\chi^2$ doesn't indicate poor fit; it indicates that you have rejected the null of perfect fit (i.e., $\sum$ = $S$). Kline (2015) aside, most SEM specialists (e.g., Brown, 2006; Finch & French, 2015; Hu & Bentler, 1995; Little, 2013; MacCallum & Austin, 2000; West et al. 2012) do not seem to recommend that you put a great deal of stock in this test, if only (amongst other reasons) because many find the null hypothesis a pretty ridiculous aspiration.

Secondly, be careful not to get sucked into specifying correlated error terms willy-nilly, just on account of the mod indexes. Many reviewers are savvy enough to see this for what it is: post-hoc fit chasing. If you have good a priori reason for specifying these, great, but if not, you might reconsider (and as you will see, your fit might not be as bad as you originally feared).

Now to your questions:

1. No, not necessarily. For one, there are a plurality of recommended cutoffs for fit indexes (see Little, 2013, for some alternatives). Also, it's not as though your model fits horribly by all standards; sure, the RMSEA/SRMR aren't great, but your CFI looks quite strong. Finally (and most importantly), you should interpret your model fit indexes in context of your average standardized loading values for your model. McNeish et al. (2018) have a nice (and recent) review paper on the "reliability paradox", in which the Hu and Bentler (1999) cutoffs penalize models with more reliable indicators, and are too liberal for models with less reliable indicators. As the Hu and Bentler (1999) cutoffs were normed on average standardized loadings of ~.75, if your typical loading value is north of that, there's perhaps an argument to be made that your model fit is much better than it appears at first.
2. CFI is a relative index of model fit, whereas RMSEA and SRMSR are absolute indexes of model fit. The former appraises the improvement in fit of the current model relative to a deliberately poor model (most often, a "null" model which specifies that all observed variables are uncorrelated with one another), whereas the latter appraises the appraises the fit of the current model against a perfect fitting model. With a great CFI and questionable RMSEA/SRMR (though be mindful of the reliability paradox, described above), I would wager that the null model in your case is fitting especially poorly, making your model like great by comparison, yet in an absolute sense it still might have a ways to go before it approaches "perfect" fitting.
3. As my initial cautioning betrays, I actually wouldn't recommend that you refine the model much further, especially if you've already played around with mod indexes. What you might consider is abandoning a confirmatory specification of your model, and instead consider a framework like exploratory structural equation modelling (ESEM, Asparouhov & Muthén, 2009), where you can let the data guide your measurement model more explicitly, but still can model structural relations among latent (exploratory) variables.

References

Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16(3), 397-438.

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.

Finch, W. H., & French, B. F. (2015). Latent variable modeling with R. New York, NY: Routledge.

Hu, L., & Bentler, P. M. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 76-99). Thousand Oaks, CA: Sage.

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55.

Kline, R. B. (2015). Principles and practice of structural equation modeling. New York, NY: Guilford Press.

Little, T. D. (2013). Longitudinal structural equation modelling. New York, NY: Guilford Press.

MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201-226.

McNeish, D., An, J., & Hancock, G. R. (2018). The thorny relation between measurement quality and fit index cutoffs in latent variable models. Journal of Personality Assessment, 100(1), 43-52.

West. S. G., Taylor, A. B., & Wu, W. (2012). Model fit and model selection in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 209-231). New York, NY: Guilford Press.