# What to do when CFA fit for multi-item scale is bad?

I am not sure how to proceed with this CFA im doing in lavaan. I have a sample of 172 participants (I know that’s not much for a CFA) and 28 items with 7-point Likert scales that should load on seven factors. I did a CFA with „mlm“-estimators, but the model fit was really bad (χ2(df=329)=739.36; comparative fit index (CFI)= .69; standardized root mean square residual (SRMR)=.10; root mean square error of approximation (RMSEA)=.09; RMSEA 90% confidence interval (CI) = [.08, .10]).

I have tried the following:

• bifactor model with one general method factor —> did not converge.

• estimators for ordinal data („WLSMV“) —> Model fit: (χ2(df=329)=462; comparative fit index (CFI)= .81; standardized root mean square residual (SRMR)=.09; root mean square error of approximation (RMSEA)=.05; RMSEA 90% confidence interval (CI) = [.04, .06])

• reducing the model by items that load low on a factor and add covariances among specific items --> Model fit: χ2(df=210)=295; comparative fit index (CFI)= .86; standardized root mean square residual (SRMR)=.08; root mean square error of approximation (RMSEA)=.07; RMSEA 90% confidence interval (CI) = [.06, .08].

Now my questions:

• What should I do with such a model?

• What would be statistically correct to do?

• Report that it does fit or that it does not fit? And which of those models?

Here is the lavaan output of the CFA of the original model:

    lavaan (0.5-17.703) converged normally after  55 iterations

Used       Total
Number of observations                           149         172

Estimator                                         ML      Robust
Minimum Function Test Statistic              985.603     677.713
Degrees of freedom                               329         329
P-value (Chi-square)                           0.000       0.000
Scaling correction factor                                  1.454
for the Satorra-Bentler correction

Model test baseline model:

Minimum Function Test Statistic             2461.549    1736.690
Degrees of freedom                               378         378
P-value                                        0.000       0.000

User model versus baseline model:

Comparative Fit Index (CFI)                    0.685       0.743
Tucker-Lewis Index (TLI)                       0.638       0.705

Loglikelihood and Information Criteria:

Loglikelihood user model (H0)              -6460.004   -6460.004
Loglikelihood unrestricted model (H1)      -5967.202   -5967.202

Number of free parameters                        105         105
Akaike (AIC)                               13130.007   13130.007
Bayesian (BIC)                             13445.421   13445.421
Sample-size adjusted Bayesian (BIC)        13113.126   13113.126

Root Mean Square Error of Approximation:

RMSEA                                          0.116       0.084
90 Percent Confidence Interval          0.107  0.124       0.077  0.092
P-value RMSEA <= 0.05                          0.000       0.000

Standardized Root Mean Square Residual:

SRMR                                           0.096       0.096

Parameter estimates:

Information                                 Expected
Standard Errors                           Robust.sem

Estimate  Std.err  Z-value  P(>|z|)   Std.lv  Std.all
Latent variables:
IC =~
PTRI_1r           1.000                               1.093    0.691
PTRI_7            1.058    0.118    8.938    0.000    1.156    0.828
PTRI_21           0.681    0.142    4.793    0.000    0.744    0.582
PTRI_22           0.752    0.140    5.355    0.000    0.821    0.646
IG =~
PTRI_10           1.000                               0.913    0.600
PTRI_11r          0.613    0.152    4.029    0.000    0.559    0.389
PTRI_19           1.113    0.177    6.308    0.000    1.016    0.737
PTRI_24           0.842    0.144    5.854    0.000    0.769    0.726
DM =~
PTRI_15r          1.000                               0.963    0.673
PTRI_16           0.892    0.118    7.547    0.000    0.859    0.660
PTRI_23           0.844    0.145    5.817    0.000    0.813    0.556
PTRI_26           1.288    0.137    9.400    0.000    1.240    0.887
IM =~
PTRI_13           1.000                               0.685    0.609
PTRI_14           1.401    0.218    6.421    0.000    0.960    0.814
PTRI_18           0.931    0.204    4.573    0.000    0.638    0.604
PTRI_20r          1.427    0.259    5.514    0.000    0.978    0.674
IN =~
PTRI_2            1.000                               0.839    0.612
PTRI_6            1.286    0.180    7.160    0.000    1.080    0.744
PTRI_12           1.031    0.183    5.644    0.000    0.866    0.523
PTRI_17r          1.011    0.208    4.872    0.000    0.849    0.613
EN =~
PTRI_3            1.000                               0.888    0.687
PTRI_8            1.136    0.146    7.781    0.000    1.008    0.726
PTRI_25           0.912    0.179    5.088    0.000    0.810    0.620
PTRI_27r          1.143    0.180    6.362    0.000    1.015    0.669
RM =~
PTRI_4r           1.000                               1.114    0.700
PTRI_9            0.998    0.105    9.493    0.000    1.112    0.786
PTRI_28           0.528    0.120    4.403    0.000    0.588    0.443
PTRI_5            0.452    0.149    3.037    0.002    0.504    0.408

Covariances:
IC ~~
IG                0.370    0.122    3.030    0.002    0.371    0.371
DM                0.642    0.157    4.075    0.000    0.610    0.610
IM                0.510    0.154    3.308    0.001    0.681    0.681
IN                0.756    0.169    4.483    0.000    0.824    0.824
EN                0.839    0.169    4.979    0.000    0.865    0.865
RM                0.644    0.185    3.479    0.001    0.529    0.529
IG ~~
DM                0.380    0.103    3.684    0.000    0.433    0.433
IM                0.313    0.096    3.248    0.001    0.501    0.501
IN                0.329    0.107    3.073    0.002    0.429    0.429
EN                0.369    0.100    3.673    0.000    0.455    0.455
RM                0.289    0.116    2.495    0.013    0.284    0.284
DM ~~
IM                0.530    0.120    4.404    0.000    0.804    0.804
IN                0.590    0.122    4.839    0.000    0.731    0.731
EN                0.588    0.105    5.619    0.000    0.688    0.688
RM                0.403    0.129    3.132    0.002    0.376    0.376
IM ~~
IN                0.439    0.126    3.476    0.001    0.763    0.763
EN                0.498    0.121    4.128    0.000    0.818    0.818
RM                0.552    0.122    4.526    0.000    0.723    0.723
IN ~~
EN                0.735    0.167    4.402    0.000    0.987    0.987
RM                0.608    0.141    4.328    0.000    0.650    0.650
EN ~~
RM                0.716    0.157    4.561    0.000    0.724    0.724

Variances:
PTRI_1r           1.304    0.272                      1.304    0.522
PTRI_7            0.613    0.153                      0.613    0.314
PTRI_21           1.083    0.199                      1.083    0.662
PTRI_22           0.940    0.141                      0.940    0.582
PTRI_10           1.483    0.257                      1.483    0.640
PTRI_11r          1.755    0.318                      1.755    0.849
PTRI_19           0.868    0.195                      0.868    0.457
PTRI_24           0.530    0.109                      0.530    0.473
PTRI_15r          1.121    0.220                      1.121    0.547
PTRI_16           0.955    0.200                      0.955    0.564
PTRI_23           1.475    0.219                      1.475    0.691
PTRI_26           0.417    0.120                      0.417    0.213
PTRI_13           0.797    0.113                      0.797    0.629
PTRI_14           0.468    0.117                      0.468    0.337
PTRI_18           0.709    0.134                      0.709    0.635
PTRI_20r          1.152    0.223                      1.152    0.546
PTRI_2            1.178    0.251                      1.178    0.626
PTRI_6            0.942    0.191                      0.942    0.447
PTRI_12           1.995    0.235                      1.995    0.727
PTRI_17r          1.199    0.274                      1.199    0.625
PTRI_3            0.882    0.179                      0.882    0.528
PTRI_8            0.910    0.131                      0.910    0.472
PTRI_25           1.048    0.180                      1.048    0.615
PTRI_27r          1.273    0.238                      1.273    0.553
PTRI_4r           1.294    0.242                      1.294    0.510
PTRI_9            0.763    0.212                      0.763    0.382
PTRI_28           1.419    0.183                      1.419    0.804
PTRI_5            1.269    0.259                      1.269    0.833
IC                1.194    0.270                      1.000    1.000
IG                0.833    0.220                      1.000    1.000
DM                0.927    0.181                      1.000    1.000
IM                0.470    0.153                      1.000    1.000
IN                0.705    0.202                      1.000    1.000
EN                0.788    0.177                      1.000    1.000
RM                1.242    0.257                      1.000    1.000

• I have the impression that the data simply do not conform to the model, e.g., you have some extremely high correlations between the factors. Try to look at a standardized solution to get correlations instead of covariances (and at standardized loadings, too). Maybe you want to collapse some factors? Maybe you want to add a method factor for the reverse-coded items if you have some--that often improves fit considerably. – hplieninger Aug 14 '14 at 8:48
• I have already tried considering the reverse-coded items with a method factor. Improved the fit, but not by much. I would like to collapse a factor or two, but I'm "bound" to stick with the theoretically postulated 7 factor solution. And even if I do collapse, the fit doesn't improve much. – teeglaze Aug 15 '14 at 13:06

### 1. Go back to Exploratory Factor Analysis

If you're getting very bad CFA fits, then it's often a sign that you have jumped too quickly to CFA. You should go back to exploratory factor analysis to learn about the structure of your test. If you have a large sample (in your case you don't), then you can split your sample to have an exploratory and a confirmatory sample.

• Apply exploratory factor analysis procedures to check whether the theorised number of factors seems reasonable. I'd check the scree plot to see what it suggests. I'd then check the rotated factor loading matrix with the theorised number of factors as well as with one or two more and one or two less factors. You can often see signs of under or over extraction of factors by looking at such factor loading matrices.
• Use exploratory factor analysis to identify problematic items. In particular, items loading most on a non-theorised factor, items with large cross-loadings, items that don't load highly on any factor.

The benefits of EFA is that it gives a lot of freedom, so you'll learn a lot more about the structure of the test than you will from only looking at CFA modification indices.

Anyway, hopefully from this process you may have identified a few issues and solutions. For example, you might drop a few items; you might update your theoretical model of how many factors there are and so on.

### 2. Improve the Confirmatory Factor Analysis Fit

There are many points that could be made here:

CFA on scales with many items per scale often perform poorly by traditional standards. This often leads people (and note I think this response is often unfortunate) to form item parcels or only use three or four items per scale. The problem is that typically proposed CFA structures fail to capture the small nuances in the data (e.g., small cross loadings, items within a test that correlate a little more than others, minor nuisance factors). These are amplified with many items per scale.

Here are a few responses to the above situation:

• Do exploratory SEM that allows for various small cross-loadings and related terms
• Examine modification indices and incorporate some of the largest reasonable modifications; e.g., a few within scale correlated residuals; a few cross-loadings. see modificationindices(fit) in lavaan.
• Use item parcelling to reduce the number of observed variables

• thanks a lot for your advices. I have already returned to EFA, but with your suggestions I figured out that a lot of items don't load on the factor they should. I am tempted to collapse the model to 5 factors instead of 7 theoretical factors, but my Prof wouldn't agree on that, but that's fine. Sadly, the 7 factor model with 4 items each doesn't work out (no matter what is modified). I will report a reduced CFA (with 7 factors + 1 bifactor, 3 items each), that barely doesn't fit (CFI=.89, RMSEA=.067, SRMR=.069), but it's the best I got. – teeglaze Aug 15 '14 at 13:00
• p.s. Jeromy, I really like your blog. It has helped me a lot so far and will certainly do so in the future :) Thanks! – teeglaze Aug 15 '14 at 13:01

I would work on trying to get the bifactor model to converge. Try adjusting the starting values...this may be a fishy approach though, so bear that in mind and interpret with caution. Read up on the dangers of interpreting models that resist convergence if you want to be truly cautious – I admit I haven't done this much yet myself in my study of SEM, so I suggest doing what you need to do to get the model to converge mostly for your benefit. I don't know that it will be any more suitable for publication, but if it clearly isn't because the bifactor model doesn't fit well either, that might be good for you to know.

Otherwise, it seems like you've done about as much as you can with the data you have. AFAIK (I've been looking deeply into this lately for a methodological project of my own, so please correct me if I'm wrong!!), WLSMV estimation in lavaan uses thresholds from polychoric correlations, which is the best way to get good fit indices out of a CFA of ordinal data. Assuming you've specified your model correctly (or at least optimally), that's about all you can do. Removing items with low loadings and freely estimating inter-item covariances is even going a bit far, but you tried that too.

Your model does not fit well by conventional standards, as you're probably aware. Of course you should not say it fits well when it doesn't. This applies to all sets of fit statistics you report here, unfortunately (I assume you were hoping it would fit). Some of your fit statistics are only fairly poor, not outright bad (the RMSEA = .05 is acceptable), but overall, none of it is good news, and you have a responsibility to be honest about that if you're going to publish these results. I hope you can, FWIW.

Either way, you might consider collecting more data if you can; that could help, depending on what you're after. If your objective is a confirmatory hypothesis test, well, you've "peeked" at your data, and will inflate your error rate if you reuse it in an expanded sample, so unless you can just set this dataset aside and replicate a whole, fresh, larger one, you've got a tough scenario to handle. If you're mostly interested in estimating parameters and narrowing confidence intervals though, I think it might be reasonable to just pool as much data as you can gather, including any you've already used here. If you can get more data, you may get better fit indices, which would make your parameter estimates more reliable. Hopefully that's good enough.

• Big +1 for @Jeromy's alternative too: go back to EFA. Exploratory bifactor analysis is an option too. There's even a couple articles on exploratory SEM (which he also mentions!) out there that I still need to read...again, these aren't quite CFA in the way you might want, but if your objectives suit these methods, your options might not be exhausted yet after all. – Nick Stauner Aug 14 '14 at 8:51
• The bifactor model converges when removing one item. But the fit is still really bad and the factors still correlate highly. I think my options are exhausted after all. However, we are collecting more data to have more reliable estimates. Thanks for your reply! – teeglaze Aug 15 '14 at 13:06