When can Model fit indices (RMSE, CFI, TLI) be trusted?

Question

I ran a three-level SEM and I have got perfect Model fit indices (RMSE=0, CFI=1, TLI=1). Data structure is households (hhid) nested inside the clusters (vilid), and being measured twice. The number of observations is 1160 (600 households but being measured twice). The number of clusters is 38. I am using Mplus to estimate the model.

The model is:

      %WITHIN%  
        x1 ON y1;   
        x2 ON y1 y2;
        y2 ON y1 y3;
        y3 ON y1 x2;
        x2 WITH x1; 
        x2 WITH y3; 
        
        %BETWEEN hhid%  
        x2 ON y4 y5 y7;
        x1 ON x2 y4 y5 y6 y7;
        
        %BETWEEN vilid% 
        x2 ON y8 x3;
        x1 ON x3 x4;
        x3 ON y8;   
        x4 ON x3;   
        x4 WITH x3;

My model still has 6 degrees of freedom, and as I understand, my model is not just identified model. However, I still have some insignificant paths. In this case, can I trust those indices? And a more general question, when these indices can be trusted?

 Chi-Square Test of Model Fit
      Value                              3.730*
      Degrees of Freedom                     6
      P-Value                           0.7131
      Scaling Correction Factor         1.1309
        for MLR
     *      The chi-square value for MLM, MLMV, MLR, ULSMV, WLSM and WLSMV cannot be used for 
           chi-square difference testing in the regular way.  MLM, MLR and WLSM 
          chi-square difference testing is described on the Mplus website.  MLMV, WLSMV, and
          ULSMV difference testing is done using the DIFFTEST option.

RMSEA (Root Mean Square Error Of Approximation)
      Estimate                           0.000
CFI/TLI
      CFI                                1.000
      TLI                                1.000
Chi-Square Test of Model Fit for the Baseline Model
      Value                            437.192
      Degrees of Freedom                    20
      P-Value                           0.0000
SRMR (Standardized Root Mean Square Residual)

      Value for Within                   0.014
      Value for Between Level 2          0.070
      Value for Between Level 3          0.012

Answer 1

I don't think 'trust' is the right word for any model fit index.

Fit indices provide information about your model fit - and different indices provide different sorts of information (or perhaps evidence), and they indicate different sorts of problems - whether those problems concern you depends on the characteristics model you are fitting, along with characteristics of the sample, and probably also on what conclusions you would like to draw.

Looking at the model about the chi-square is low and not statistically significant. The evidence I can draw from this depends on the sample size, how much post-hoc modification you did to the model and the number of parameters estimated.

RMSEA is zero, because chi-square is lower than DF, so that didn't tell us anything we didn't already know.

The incremental fit indices look good, because the null model chi-square is high and chi-square is low. This is usually a good thing, but the caveats associated with chi-square apply.

From what I know (which isn't enough) that model fit looks pretty good. But if you told me that the sample size was 30, I might reconsider that conclusion.

Answer 2

A model with a decent chi-square can still contain non-significant paths. This does not make the model test questionable. The fact that a model fits well does not mean that it necessarily contains large effects among the variables in the model. The overall chi-square model fit test assesses whether the model-implied covariance structure matches the observed structure--regardless of whether the observed structure contains large or small correlations/effects. (Note, however, that the power of the chi-square to detect misfit is related to the magnitude of the correlations.)