# SEM: High SRMR in model with binary predictor. Other fit indices OK [closed]

In this post I may display severe lack of understanding in SEM and the math behind it so please be kind.

I am testing the following structural model with lavaan using MLM (due to non-normality) with n = 310:

sem.model ='
propensity =~ DTT_04 + DTT_01 + DTT_02 + DTT_03
ability =~ ABI_02 + ABI_01 + ABI_03
benint =~ BEN_04 + BEN_03 + BEN_05 + INT_01 + INT_03 + INT_04 + INT_05 + INT_06
trust =~ TRU_02 + TRU_03 + TRU_04
experience =~ MOT_X_01 + MOT_X_02 + MOT_X_03 + MOT_X_04 + MOT_X_05
money =~ MOT_M_01 + MOT_M_02 + MOT_M_03 + MOT_M_04 + MOT_M_05

experience ~ GROUP
money ~ GROUP
benint ~ experience
benint ~ money
trust ~ ability
trust ~ benint
trust ~ propensity

'


GROUP is a directly observed binary predictor (0,1) indicating control group or treatment group (groups have approximately the same size). All other variables are continuous (7-point Likert scale).

Results of a CFA indicated good fit of the measurement model (χ2/df = 1.44 ; CFI = 0.981 ; RMSEA = 0.039 ; SRMR = 0.036). (GROUP wasn't included in the CFA as it is directly observed).

When I do the structural model including GROUP as exogenous predictor, all of a sudden the SRMR is terribly high:

lavaan 0.6-5 ended normally after 61 iterations

Estimator                                         ML
Optimization method                           NLMINB
Number of free parameters                         64

Number of observations                           310

Model Test User Model:
Standard      Robust
Test Statistic                               830.768     798.760
Degrees of freedom                               370         370
P-value (Chi-square)                           0.000       0.000
Scaling correction factor                                  1.040
for the Satorra-Bentler correction

Model Test Baseline Model:

Test statistic                              8462.728    7556.330
Degrees of freedom                               406         406
P-value                                        0.000       0.000
Scaling correction factor                                  1.120

User Model versus Baseline Model:

Comparative Fit Index (CFI)                    0.943       0.940
Tucker-Lewis Index (TLI)                       0.937       0.934

Robust Comparative Fit Index (CFI)                         0.944
Robust Tucker-Lewis Index (TLI)                            0.939

Loglikelihood and Information Criteria:

Loglikelihood user model (H0)             -12662.869  -12662.869
Loglikelihood unrestricted model (H1)     -12247.485  -12247.485

Akaike (AIC)                               25453.738   25453.738
Bayesian (BIC)                             25692.879   25692.879
Sample-size adjusted Bayesian (BIC)        25489.895   25489.895

Root Mean Square Error of Approximation:

RMSEA                                          0.063       0.061
90 Percent confidence interval - lower         0.058       0.055
90 Percent confidence interval - upper         0.069       0.067
P-value RMSEA <= 0.05                          0.000       0.001

Robust RMSEA                                               0.062
90 Percent confidence interval - lower                     0.056
90 Percent confidence interval - upper                     0.068

Standardized Root Mean Square Residual:

SRMR                                           0.210       0.210

Parameter Estimates:

Information                                 Expected
Information saturated (h1) model          Structured
Standard errors                           Robust.sem

Latent Variables:
Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
propensity =~
DTT_04            1.000                               1.463    0.907
DTT_01            0.854    0.040   21.554    0.000    1.250    0.823
DTT_02            0.887    0.041   21.635    0.000    1.298    0.840
DTT_03            0.988    0.042   23.818    0.000    1.447    0.893
ability =~
ABI_02            1.000                               1.540    0.959
ABI_01            0.915    0.027   34.467    0.000    1.409    0.904
ABI_03            0.982    0.026   37.687    0.000    1.513    0.940
benint =~
BEN_04            1.000                               1.337    0.846
BEN_03            0.785    0.068   11.468    0.000    1.050    0.614
BEN_05            0.847    0.042   20.150    0.000    1.132    0.761
INT_01            0.764    0.050   15.168    0.000    1.021    0.740
INT_03            0.937    0.047   19.790    0.000    1.253    0.833
INT_04            0.818    0.051   16.204    0.000    1.094    0.797
INT_05            0.917    0.052   17.701    0.000    1.226    0.786
INT_06            0.983    0.046   21.405    0.000    1.314    0.859
trust =~
TRU_02            1.000                               1.405    0.909
TRU_03            0.927    0.046   20.333    0.000    1.302    0.895
TRU_04            0.933    0.052   18.041    0.000    1.311    0.832
experience =~
MOT_X_01          1.000                               1.398    0.836
MOT_X_02          0.912    0.057   15.962    0.000    1.275    0.853
MOT_X_03          0.911    0.061   15.048    0.000    1.274    0.848
MOT_X_04          0.974    0.058   16.749    0.000    1.361    0.848
MOT_X_05          0.898    0.057   15.804    0.000    1.255    0.814
money =~
MOT_M_01          1.000                               1.719    0.940
MOT_M_02          0.997    0.024   42.426    0.000    1.715    0.947
MOT_M_03          0.890    0.035   25.438    0.000    1.531    0.798
MOT_M_04          0.923    0.029   32.242    0.000    1.587    0.913
MOT_M_05          0.988    0.027   36.973    0.000    1.699    0.929

Regressions:
Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
experience ~
GROUP            -0.107    0.164   -0.651    0.515   -0.077   -0.038
money ~
GROUP             0.436    0.198    2.202    0.028    0.254    0.127
benint ~
experience        0.636    0.052   12.118    0.000    0.665    0.665
money            -0.132    0.035   -3.785    0.000   -0.170   -0.170
trust ~
ability           0.407    0.046    8.936    0.000    0.447    0.447
benint            0.347    0.054    6.448    0.000    0.330    0.330
propensity        0.115    0.051    2.274    0.023    0.120    0.120

Covariances:
Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
propensity ~~
ability           0.693    0.136    5.113    0.000    0.308    0.308

Variances:
Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
.DTT_04            0.463    0.065    7.104    0.000    0.463    0.178
.DTT_01            0.744    0.114    6.545    0.000    0.744    0.322
.DTT_02            0.701    0.106    6.594    0.000    0.701    0.294
.DTT_03            0.532    0.081    6.552    0.000    0.532    0.203
.ABI_02            0.209    0.049    4.272    0.000    0.209    0.081
.ABI_01            0.442    0.054    8.230    0.000    0.442    0.182
.ABI_03            0.300    0.062    4.841    0.000    0.300    0.116
.BEN_04            0.708    0.087    8.116    0.000    0.708    0.284
.BEN_03            1.818    0.186    9.751    0.000    1.818    0.622
.BEN_05            0.930    0.096    9.664    0.000    0.930    0.421
.INT_01            0.860    0.091    9.434    0.000    0.860    0.452
.INT_03            0.691    0.081    8.576    0.000    0.691    0.306
.INT_04            0.687    0.089    7.708    0.000    0.687    0.365
.INT_05            0.929    0.131    7.075    0.000    0.929    0.382
.INT_06            0.612    0.074    8.244    0.000    0.612    0.262
.TRU_02            0.412    0.086    4.789    0.000    0.412    0.173
.TRU_03            0.421    0.083    5.058    0.000    0.421    0.199
.TRU_04            0.766    0.128    5.975    0.000    0.766    0.308
.MOT_X_01          0.845    0.111    7.635    0.000    0.845    0.302
.MOT_X_02          0.611    0.095    6.437    0.000    0.611    0.273
.MOT_X_03          0.635    0.095    6.717    0.000    0.635    0.281
.MOT_X_04          0.723    0.124    5.829    0.000    0.723    0.281
.MOT_X_05          0.805    0.148    5.450    0.000    0.805    0.338
.MOT_M_01          0.393    0.053    7.363    0.000    0.393    0.117
.MOT_M_02          0.341    0.057    5.993    0.000    0.341    0.104
.MOT_M_03          1.334    0.172    7.768    0.000    1.334    0.363
.MOT_M_04          0.501    0.071    7.102    0.000    0.501    0.166
.MOT_M_05          0.457    0.081    5.637    0.000    0.457    0.137
propensity        2.142    0.169   12.702    0.000    1.000    1.000
ability           2.373    0.171   13.859    0.000    1.000    1.000
.benint            0.944    0.106    8.915    0.000    0.528    0.528
.trust             1.271    0.127   10.013    0.000    0.644    0.644
.experience        1.952    0.201    9.692    0.000    0.999    0.999
.money             2.908    0.211   13.757    0.000    0.984    0.984

R-Square:
Estimate
DTT_04            0.822
DTT_01            0.678
DTT_02            0.706
DTT_03            0.797
ABI_02            0.919
ABI_01            0.818
ABI_03            0.884
BEN_04            0.716
BEN_03            0.378
BEN_05            0.579
INT_01            0.548
INT_03            0.694
INT_04            0.635
INT_05            0.618
INT_06            0.738
TRU_02            0.827
TRU_03            0.801
TRU_04            0.692
MOT_X_01          0.698
MOT_X_02          0.727
MOT_X_03          0.719
MOT_X_04          0.719
MOT_X_05          0.662
MOT_M_01          0.883
MOT_M_02          0.896
MOT_M_03          0.637
MOT_M_04          0.834
MOT_M_05          0.863
benint            0.472
trust             0.356
experience        0.001
money             0.016


Because of the high SRMR, I looked at the residuals with resid(sem.fit, type="standardized")\$cov. Indeed, many of them are pretty large (.2 up to .6). When I remove GROUP from the structural model SRMR drops dramatically to .075 with none of the residuals even reaching .1.

When I analyse the parts of the structural model separately (Part 1, 2 ,3 as indicated in the diagram), fit is good for the separate parts of the model. So it really seems to be an issue of the overall model.

Questions:

1. How come residuals "explode" when I include GROUP? Can this be avoided? Could it be because GROUP is binary? Am I missing something important code-wise in that regard?
2. What can I do to improve model fit? I have already looked at modification indices but none of them make sense from a theoretical point of view. Also, so many residuals are extremely large when including GROUP in the model that I can not remove all of the corresponding variables.
3. What is the best way to proceed from here? The 𝑅2 of experience and money are pretty low. That would be an indicator of GROUP just not being a very good predictor. Is it justified then to just remove GROUP from the model and say that the original theoretical model just didn't fit?

I do not think that GROUP being binary is itself the issue, but GROUP may be contributing to the worsening of fit between CFA (factor covariances unstructured) and the structural model. GROUP could be contributing in two ways.

First, your model implies that all covariance between Experience and Money is due to those factors' joint dependence on GROUP--and yet, those predictive paths are near 0. This forces Experience and Money to be essentially uncorrelated in the structural model. You could address this by allowing residual covariance between Experience and Money, besides the covariance due to joint dependence on GROUP.

Second, GROUP is only permitted to covary with the other factors through its predictive impact on Experience and Money, and, again, those paths are near 0. You might get a view on this by re-estimating teh model without GROUP at all but with free covariance between Experience and Money. I also might suggest that you re-run the CFA but include GROUP as a single predictor of a "GROUP" factor. Those results will show you whether this variable has strong covariance with any other factor in the model. You could also look at modification indices for the structural model.

Broadly, the worsening of fit means that the constraints on factor covariances implied by the structural model is not entirely consistent with the data.

I might have suggested structuring this as a multiple group model, with the groups defined by GROUP. That would get binary GROUP out of your hair, statistically. But your sample size is small for that. Still, you might well impose measurement invariance across the two subsamples, and that would sharply cut the number of additional parameters you would be estimating. You could still capture the impact of GROUP on Experience and Money by looking at differences in their factor means.

• Thank you for your answer. I have already run CFA with GROUP as LV. Its covariance with other LVs is almost non-existent: 0.016 0.002 -0.057 -0.045 -0.024 0.109 0.250. So that means, that group simply does not covary with any other variable and is therefore not a suitable predictor? – Felix Krauth Jan 18 at 16:39
• Whoa, whoa. If these are covariances, it is hard to tell what is small. Those last two numbers may well be not small. GROUP might also serve as a moderator--something that you might see in a multi-group analysis. I am assuming that GROUP is fairly balanced, with about half of respondents in each condition. Lack of balance could contribute to low predictive power. Also, I should have added that yes, SRMR seems disproportionate to the values of the other fit indices. – Ed Rigdon Jan 18 at 16:44
• Indeed GROUP is split 50/50. I will try a multi-group model and see where that gets me. Thank you for the suggestion. – Felix Krauth Jan 18 at 16:50