What does it mean when a CFA model improves drastically by ommiting a factor?

Question

I am trying to fit a CFA model with four factors (say F1 with 4 indicators, F2 with 6 indicators, F3 with 3 indicators and F4 with 5 indicators) in Amos. Each indicator is computed by summation of some Likert scale items of the questionnaire. The sample size is $n = 451$, and the indicators have an approximately normal distribution. No sign of collinearity has been observed. The fit indices after deletion of 4 non-significant/low-factor loading indicators are:

• CMIN/df = 7.824
• CFI = .913
• TLI = .889
• NFI = .902
• RMSEA = .123
• SRMR = .060.

However, I found out that by deletion of F3, the fit indices change to:

• CMIN/df = 4.054
• CFI = .972
• TLI = .962
• NFI =.963
• RMSEA =.082
• SRMR =.033

All the F3 indicators have factor loading values greater than .7 with small standard errors. How this can be interpreted?

Answer 1

First, it could be that the additional factor in your model was so poorly fit (perhaps because of some nonlinearity present, bimodalism present from confounding categorical variables, etc.), that any model representing the data would look bad. I seems you have already attempted to look at some of these elements based off the checks you have done, but I highly stress visualizing every element of your model to ensure this is not the case.

Second, it could be that your indicators are just poorly written and caused issues during fitting. For example, double-barreled statements and vague wording can cause problems with a proper solution. If we have a question about anxiety such as "Do you feel anxious and angry when commanded by your boss at work?", we are essentially tapping into two latent factors, anxiety and anger, in the same question. Thus this could contribute to problems in your model.

Third, it could be that your perceived model simply doesn't exist, and a three-factor solution simply describes the phenomenon in question better. As an example, we could come up with a model that explains ten different types of motivation (e.g. intrinsic, extrinsic, etc.). But it may be the case that a general model of motivation simply describes the data better because splitting hairs is worse than having a generic causal influence that better explains what is going on.