I am running a confirmatory factor analysis on a very simple model. It is intended to confirm the structure of a brief rating scale for the evaluation of one social emotional domain. As usual, I split the sample (N = 307) in two parts (i.e., calibration and validation samples) and tested the same model (one-factor solution) on the two sub-samples separately and then re-tested it on the full sample. The fit indices looked very good for the calibration sample, but then when I re-ran the same analysis on the validation sample I obtained these (weird?) values. In particular, chi-square is highly nonsign., CFI is equal to 1, TLI greater than 1 (which I know it is mathematically possible but it never happened to me nor I have ever seen it in other publications), and the RMSEA is equal to zero. I also obtained similar values when I tested the model on the full sample. Am I missing anything? Or I can go ahead and write up the results of the manuscript? Are there other parts of the model I can check to make sure I am not running into some issues?

Thanks to anyone who will take the time to help!

lavaan 0.6-8 ended normally after 32 iterations

  Estimator                                         ML
  Optimization method                           NLMINB
  Number of model parameters                         8
  Number of observations                           152
Model Test User Model:
                                               Standard      Robust
  Test Statistic                                  0.413       0.238
  Degrees of freedom                                  2           2
  P-value (Chi-square)                            0.813       0.888
  Scaling correction factor                                   1.739
       Yuan-Bentler correction (Mplus variant)                     

Model Test Baseline Model:

  Test statistic                               229.862     106.486
  Degrees of freedom                                 6           6
  P-value                                        0.000       0.000
  Scaling correction factor                                  2.159

User Model versus Baseline Model:

  Comparative Fit Index (CFI)                    1.000       1.000
  Tucker-Lewis Index (TLI)                       1.021       1.053
  Robust Comparative Fit Index (CFI)                         1.000
  Robust Tucker-Lewis Index (TLI)                            1.042

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)               -940.922    -940.922
  Scaling correction factor                                  2.458
      for the MLR correction                                      
  Loglikelihood unrestricted model (H1)       -940.716    -940.716
  Scaling correction factor                                  2.314
      for the MLR correction                                      
  Akaike (AIC)                                1897.845    1897.845
  Bayesian (BIC)                              1922.036    1922.036
  Sample-size adjusted Bayesian (BIC)         1896.716    1896.716

Root Mean Square Error of Approximation:

  RMSEA                                          0.000       0.000
  90 Percent confidence interval - lower         0.000       0.000
  90 Percent confidence interval - upper         0.098       0.000
  P-value RMSEA <= 0.05                          0.867       0.981
  Robust RMSEA                                               0.000
  90 Percent confidence interval - lower                     0.000
  90 Percent confidence interval - upper                     0.000

Standardized Root Mean Square Residual:

  SRMR                                           0.010       0.010

Parameter Estimates:

  Standard errors                             Sandwich
  Information bread                           Observed
  Observed information based on                Hessian

Latent Variables:
                     Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
  F1 =~                                                                 
    Item33              1.000                               0.599    0.450
    Item35              2.275    0.622    3.658    0.000    1.362    0.943
    Item36              1.841    0.481    3.829    0.000    1.102    0.675
    Item37              1.473    0.346    4.261    0.000    0.882    0.778

                     Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
   .Item33              1.414    0.240    5.886    0.000    1.414    0.798
   .Item35              0.231    0.204    1.131    0.258    0.231    0.111
   .Item36              1.453    0.250    5.802    0.000    1.453    0.545
   .Item37              0.506    0.138    3.665    0.000    0.506    0.394
    F1                  0.358    0.204    1.758    0.079    1.000    1.000

You're not missing anything, you have a very good fit.

You only have four items in the scale, so it's not that hard to get a good fit. Some of your items appear to be very highly correlated - this increases the chi-square of the null model, which increases TLI and CFI.

Your chi-square is lower than df, so your RMSEA will be zero (look at the formula). The RMSEA has quite a wide CI, but that's not really an issue.

Your scaling correction factor is nearly 2 - this is pretty high, suggesting you have high (multivariate) kurtosis. Are these categorical items with a small (ish) number of categories? It might be worth trying the WLSMV estimator.

  • $\begingroup$ Thanks so much for your detailed answer, Jeremy! You are absolutely right. Almost all the items were rated on a 7-level Likert scale which described a behavioral domain with low base rate (hence with some degree of skewness and very high kurtosis). I though the "MLR" estimator would take care of this feature, but clearly it did not. Can you tell me what is generally an acceptable range for the scaling correction factor? Additionally, I re-run the analyses using the WLSMV estimator and the correction factor associated with the chi-square (I hope you referred to this) dropped to 0.168. $\endgroup$ Apr 25 at 4:43
  • $\begingroup$ A 7 point scale is not too bad. MLR does take care of it, it doesn't necessarily indicate a problem - I've sometimes seen this happen with 2 or 3 point scales, just making sure that wasn't the case. $\endgroup$ Apr 25 at 5:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.