What to do when CFA fit for multi-item scale is bad?

Question

I am not sure how to proceed with this CFA im doing in lavaan. I have a sample of 172 participants (I know that’s not much for a CFA) and 28 items with 7-point Likert scales that should load on seven factors. I did a CFA with „mlm“-estimators, but the model fit was really bad (χ2(df=329)=739.36; comparative fit index (CFI)= .69; standardized root mean square residual (SRMR)=.10; root mean square error of approximation (RMSEA)=.09; RMSEA 90% confidence interval (CI) = [.08, .10]).

I have tried the following:

• bifactor model with one general method factor —> did not converge.

• estimators for ordinal data („WLSMV“) —> Model fit: (χ2(df=329)=462; comparative fit index (CFI)= .81; standardized root mean square residual (SRMR)=.09; root mean square error of approximation (RMSEA)=.05; RMSEA 90% confidence interval (CI) = [.04, .06])

• reducing the model by items that load low on a factor and add covariances among specific items --> Model fit: χ2(df=210)=295; comparative fit index (CFI)= .86; standardized root mean square residual (SRMR)=.08; root mean square error of approximation (RMSEA)=.07; RMSEA 90% confidence interval (CI) = [.06, .08].

Now my questions:

• What should I do with such a model?

• What would be statistically correct to do?

• Report that it does fit or that it does not fit? And which of those models?

I’d be glad to have a discussion with you about this.

Here is the lavaan output of the CFA of the original model:

    lavaan (0.5-17.703) converged normally after  55 iterations

                                              Used       Total
  Number of observations                           149         172

  Estimator                                         ML      Robust
  Minimum Function Test Statistic              985.603     677.713
  Degrees of freedom                               329         329
  P-value (Chi-square)                           0.000       0.000
  Scaling correction factor                                  1.454
    for the Satorra-Bentler correction

Model test baseline model:

  Minimum Function Test Statistic             2461.549    1736.690
  Degrees of freedom                               378         378
  P-value                                        0.000       0.000

User model versus baseline model:

  Comparative Fit Index (CFI)                    0.685       0.743
  Tucker-Lewis Index (TLI)                       0.638       0.705

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)              -6460.004   -6460.004
  Loglikelihood unrestricted model (H1)      -5967.202   -5967.202

  Number of free parameters                        105         105
  Akaike (AIC)                               13130.007   13130.007
  Bayesian (BIC)                             13445.421   13445.421
  Sample-size adjusted Bayesian (BIC)        13113.126   13113.126

Root Mean Square Error of Approximation:

  RMSEA                                          0.116       0.084
  90 Percent Confidence Interval          0.107  0.124       0.077  0.092
  P-value RMSEA <= 0.05                          0.000       0.000

Standardized Root Mean Square Residual:

  SRMR                                           0.096       0.096

Parameter estimates:

  Information                                 Expected
  Standard Errors                           Robust.sem

                   Estimate  Std.err  Z-value  P(>|z|)   Std.lv  Std.all
Latent variables:
  IC =~
    PTRI_1r           1.000                               1.093    0.691
    PTRI_7            1.058    0.118    8.938    0.000    1.156    0.828
    PTRI_21           0.681    0.142    4.793    0.000    0.744    0.582
    PTRI_22           0.752    0.140    5.355    0.000    0.821    0.646
  IG =~
    PTRI_10           1.000                               0.913    0.600
    PTRI_11r          0.613    0.152    4.029    0.000    0.559    0.389
    PTRI_19           1.113    0.177    6.308    0.000    1.016    0.737
    PTRI_24           0.842    0.144    5.854    0.000    0.769    0.726
  DM =~
    PTRI_15r          1.000                               0.963    0.673
    PTRI_16           0.892    0.118    7.547    0.000    0.859    0.660
    PTRI_23           0.844    0.145    5.817    0.000    0.813    0.556
    PTRI_26           1.288    0.137    9.400    0.000    1.240    0.887
  IM =~
    PTRI_13           1.000                               0.685    0.609
    PTRI_14           1.401    0.218    6.421    0.000    0.960    0.814
    PTRI_18           0.931    0.204    4.573    0.000    0.638    0.604
    PTRI_20r          1.427    0.259    5.514    0.000    0.978    0.674
  IN =~
    PTRI_2            1.000                               0.839    0.612
    PTRI_6            1.286    0.180    7.160    0.000    1.080    0.744
    PTRI_12           1.031    0.183    5.644    0.000    0.866    0.523
    PTRI_17r          1.011    0.208    4.872    0.000    0.849    0.613
  EN =~
    PTRI_3            1.000                               0.888    0.687
    PTRI_8            1.136    0.146    7.781    0.000    1.008    0.726
    PTRI_25           0.912    0.179    5.088    0.000    0.810    0.620
    PTRI_27r          1.143    0.180    6.362    0.000    1.015    0.669
  RM =~
    PTRI_4r           1.000                               1.114    0.700
    PTRI_9            0.998    0.105    9.493    0.000    1.112    0.786
    PTRI_28           0.528    0.120    4.403    0.000    0.588    0.443
    PTRI_5            0.452    0.149    3.037    0.002    0.504    0.408

Covariances:
  IC ~~
    IG                0.370    0.122    3.030    0.002    0.371    0.371
    DM                0.642    0.157    4.075    0.000    0.610    0.610
    IM                0.510    0.154    3.308    0.001    0.681    0.681
    IN                0.756    0.169    4.483    0.000    0.824    0.824
    EN                0.839    0.169    4.979    0.000    0.865    0.865
    RM                0.644    0.185    3.479    0.001    0.529    0.529
  IG ~~
    DM                0.380    0.103    3.684    0.000    0.433    0.433
    IM                0.313    0.096    3.248    0.001    0.501    0.501
    IN                0.329    0.107    3.073    0.002    0.429    0.429
    EN                0.369    0.100    3.673    0.000    0.455    0.455
    RM                0.289    0.116    2.495    0.013    0.284    0.284
  DM ~~
    IM                0.530    0.120    4.404    0.000    0.804    0.804
    IN                0.590    0.122    4.839    0.000    0.731    0.731
    EN                0.588    0.105    5.619    0.000    0.688    0.688
    RM                0.403    0.129    3.132    0.002    0.376    0.376
  IM ~~
    IN                0.439    0.126    3.476    0.001    0.763    0.763
    EN                0.498    0.121    4.128    0.000    0.818    0.818
    RM                0.552    0.122    4.526    0.000    0.723    0.723
  IN ~~
    EN                0.735    0.167    4.402    0.000    0.987    0.987
    RM                0.608    0.141    4.328    0.000    0.650    0.650
  EN ~~
    RM                0.716    0.157    4.561    0.000    0.724    0.724


Variances:
    PTRI_1r           1.304    0.272                      1.304    0.522
    PTRI_7            0.613    0.153                      0.613    0.314
    PTRI_21           1.083    0.199                      1.083    0.662
    PTRI_22           0.940    0.141                      0.940    0.582
    PTRI_10           1.483    0.257                      1.483    0.640
    PTRI_11r          1.755    0.318                      1.755    0.849
    PTRI_19           0.868    0.195                      0.868    0.457
    PTRI_24           0.530    0.109                      0.530    0.473
    PTRI_15r          1.121    0.220                      1.121    0.547
    PTRI_16           0.955    0.200                      0.955    0.564
    PTRI_23           1.475    0.219                      1.475    0.691
    PTRI_26           0.417    0.120                      0.417    0.213
    PTRI_13           0.797    0.113                      0.797    0.629
    PTRI_14           0.468    0.117                      0.468    0.337
    PTRI_18           0.709    0.134                      0.709    0.635
    PTRI_20r          1.152    0.223                      1.152    0.546
    PTRI_2            1.178    0.251                      1.178    0.626
    PTRI_6            0.942    0.191                      0.942    0.447
    PTRI_12           1.995    0.235                      1.995    0.727
    PTRI_17r          1.199    0.274                      1.199    0.625
    PTRI_3            0.882    0.179                      0.882    0.528
    PTRI_8            0.910    0.131                      0.910    0.472
    PTRI_25           1.048    0.180                      1.048    0.615
    PTRI_27r          1.273    0.238                      1.273    0.553
    PTRI_4r           1.294    0.242                      1.294    0.510
    PTRI_9            0.763    0.212                      0.763    0.382
    PTRI_28           1.419    0.183                      1.419    0.804
    PTRI_5            1.269    0.259                      1.269    0.833
    IC                1.194    0.270                      1.000    1.000
    IG                0.833    0.220                      1.000    1.000
    DM                0.927    0.181                      1.000    1.000
    IM                0.470    0.153                      1.000    1.000
    IN                0.705    0.202                      1.000    1.000
    EN                0.788    0.177                      1.000    1.000
    RM                1.242    0.257                      1.000    1.000

Answer 1

1. Go back to Exploratory Factor Analysis

If you're getting very bad CFA fits, then it's often a sign that you have jumped too quickly to CFA. You should go back to exploratory factor analysis to learn about the structure of your test. If you have a large sample (in your case you don't), then you can split your sample to have an exploratory and a confirmatory sample.

• Apply exploratory factor analysis procedures to check whether the theorised number of factors seems reasonable. I'd check the scree plot to see what it suggests. I'd then check the rotated factor loading matrix with the theorised number of factors as well as with one or two more and one or two less factors. You can often see signs of under or over extraction of factors by looking at such factor loading matrices.
• Use exploratory factor analysis to identify problematic items. In particular, items loading most on a non-theorised factor, items with large cross-loadings, items that don't load highly on any factor.

The benefits of EFA is that it gives a lot of freedom, so you'll learn a lot more about the structure of the test than you will from only looking at CFA modification indices.

Anyway, hopefully from this process you may have identified a few issues and solutions. For example, you might drop a few items; you might update your theoretical model of how many factors there are and so on.

2. Improve the Confirmatory Factor Analysis Fit

There are many points that could be made here:

CFA on scales with many items per scale often perform poorly by traditional standards. This often leads people (and note I think this response is often unfortunate) to form item parcels or only use three or four items per scale. The problem is that typically proposed CFA structures fail to capture the small nuances in the data (e.g., small cross loadings, items within a test that correlate a little more than others, minor nuisance factors). These are amplified with many items per scale.

Here are a few responses to the above situation:

• Do exploratory SEM that allows for various small cross-loadings and related terms
• Examine modification indices and incorporate some of the largest reasonable modifications; e.g., a few within scale correlated residuals; a few cross-loadings. see modificationindices(fit) in lavaan.
• Use item parcelling to reduce the number of observed variables

General comments

So in general, if you're CFA model is really bad, return to EFA to learn more about your scale. Alternatively if your EFA is good, and your CFA just looks a little bad due to well known problems of having many items per scale, then standard CFA approaches as mentioned above are appropriate.

Answer 2

I would work on trying to get the bifactor model to converge. Try adjusting the starting values...this may be a fishy approach though, so bear that in mind and interpret with caution. Read up on the dangers of interpreting models that resist convergence if you want to be truly cautious – I admit I haven't done this much yet myself in my study of SEM, so I suggest doing what you need to do to get the model to converge mostly for your benefit. I don't know that it will be any more suitable for publication, but if it clearly isn't because the bifactor model doesn't fit well either, that might be good for you to know.

Otherwise, it seems like you've done about as much as you can with the data you have. AFAIK (I've been looking deeply into this lately for a methodological project of my own, so please correct me if I'm wrong!!), WLSMV estimation in lavaan uses thresholds from polychoric correlations, which is the best way to get good fit indices out of a CFA of ordinal data. Assuming you've specified your model correctly (or at least optimally), that's about all you can do. Removing items with low loadings and freely estimating inter-item covariances is even going a bit far, but you tried that too.

Your model does not fit well by conventional standards, as you're probably aware. Of course you should not say it fits well when it doesn't. This applies to all sets of fit statistics you report here, unfortunately (I assume you were hoping it would fit). Some of your fit statistics are only fairly poor, not outright bad (the RMSEA = .05 is acceptable), but overall, none of it is good news, and you have a responsibility to be honest about that if you're going to publish these results. I hope you can, FWIW.

Either way, you might consider collecting more data if you can; that could help, depending on what you're after. If your objective is a confirmatory hypothesis test, well, you've "peeked" at your data, and will inflate your error rate if you reuse it in an expanded sample, so unless you can just set this dataset aside and replicate a whole, fresh, larger one, you've got a tough scenario to handle. If you're mostly interested in estimating parameters and narrowing confidence intervals though, I think it might be reasonable to just pool as much data as you can gather, including any you've already used here. If you can get more data, you may get better fit indices, which would make your parameter estimates more reliable. Hopefully that's good enough.