3
$\begingroup$

I've fitted a one-factor model to data originating from a unidimensional 8-item scale; sample size is 400. Because the scale uses a Likert response format, and data does not follow a multivariate normal distribution, I have used DWLS. I have gotten non-significant results for Chi-squared, and therefore extremely good values for the rest of the fit indexes I have calculated. Since this is the first time I am seeing these type of results, I wonder: is it possible? Am I missing something? Should I instead use robust estimation methods (WLSMV)?


lavaan 0.6-5 ended normally after 29 iterations

  Estimator                                       DWLS
  Optimization method                           NLMINB
  Number of free parameters                         16

  Number of observations                           400

Model Test User Model:

  Test statistic                                16.916
  Degrees of freedom                                20
  P-value (Chi-square)                           0.658

Model Test Baseline Model:

  Test statistic                              4000.700
  Degrees of freedom                                28
  P-value                                        0.000

User Model versus Baseline Model:

  Comparative Fit Index (CFI)                    1.000
  Tucker-Lewis Index (TLI)                       1.001
$\endgroup$

3 Answers 3

4
$\begingroup$

Clearly it's possible, because it happened.

Typically when you have very low chi-squares, you sometimes have very lower CFI/TLI - that was the first think I looked at, because they indicate lower power.

You don't have low power, you just have a well fitting model. This is not a problem.

$\endgroup$
5
  • $\begingroup$ Thanks! I am so used to report a significant Chi-squared, disregard it, and then move on to other fit indexes to justify the model that I needed someone else to look at these results just to be sure :) $\endgroup$
    – Bea Hs
    Commented Dec 23, 2019 at 19:02
  • $\begingroup$ If you can add the correlation matrix, I'd be interested to see it. $\endgroup$ Commented Dec 23, 2019 at 19:08
  • 1
    $\begingroup$ It appears that you have not declared the observed variables to be ordinal, but are treating the Likert responses as if they were continuous. It is possible that you have extremely flat (platykurtic) distributions, which are suppressing the DWLS chi-square. Honestly, the results look like what I get when I simulate data for a classroom assignment. But you did say that the set of items were unidimensional, and if that is so, then these results are what you should expect. Your sample size is probably large enough for asymptotic properties to hold. $\endgroup$
    – Ed Rigdon
    Commented Dec 23, 2019 at 22:32
  • $\begingroup$ @Ed Rigdon - nice spot! $\endgroup$ Commented Dec 23, 2019 at 23:14
  • 1
    $\begingroup$ @EdRigdon Hi, I just saw your comment. You were right regarding not having declared the observed variables as ordered. I just did and I got a significant chi-square. The one-factor model still achieves an almost perfect fit to the data: CFI/TLI are now 0.99, RMSEA and SRMR are below typical threshold levels etc. so the end result is the same but I am happy I can report the correct chi-square and p-value. Thanks! $\endgroup$
    – Bea Hs
    Commented Jan 11, 2020 at 9:39
3
$\begingroup$

Note that it is very common to have a non-significant Chi-squared fit statistic in CFA testing, as it is heavily sensitive to large sample sizes and higher model complexity (i.e. a number of indicators in your model). In your case, it is very unsurprising that the Chi-squared in non-significant, but both CFI and TLI are large. I will gently echo @JeremyMiles conclusion that you have a well-fitting model. Currently, in much of psychometric and social sciences CFA-based research, the Chi-squared is only reported for historical reasons and it is not used as a decisive fit statistic.

I think you will massively benefit from this thread, as it provides you a somewhat succinct summary of each fit statistic, and most importantly for your case, the table of which CFA fit statistic is sensitive to what model condition.

$\endgroup$
0
$\begingroup$

DWLS can inflate RMSEA, CFI and TLI (Xia and Yang, 2018). The SRMR seems to perform better (Shi et al. 2020).

If the SRMR is bad, the good CFI and TLI may be "fake news".

However, the fit may actually be good, but the DWLS estimator is pushing the fit indices into "too good to be true" territory.

Personally, I would also run the analysis treating the items as continuous, i.e. MLM(VS) or MLR estimator, if the items have 5 or more levels. If the fit indices are much worse in this scenario, that would indicate to me that the fit indices from the DWLS estimator are not to be trusted.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.