0
$\begingroup$

In this post I may display severe lack of understanding in SEM and the math behind it so please be kind.

I am testing the following structural model with lavaan using MLM (due to non-normality) with n = 310:

sem.model ='
propensity =~ DTT_04 + DTT_01 + DTT_02 + DTT_03
ability =~ ABI_02 + ABI_01 + ABI_03
benint =~ BEN_04 + BEN_03 + BEN_05 + INT_01 + INT_03 + INT_04 + INT_05 + INT_06
trust =~ TRU_02 + TRU_03 + TRU_04
experience =~ MOT_X_01 + MOT_X_02 + MOT_X_03 + MOT_X_04 + MOT_X_05
money =~ MOT_M_01 + MOT_M_02 + MOT_M_03 + MOT_M_04 + MOT_M_05


experience ~ GROUP
money ~ GROUP
benint ~ experience
benint ~ money
trust ~ ability
trust ~ benint
trust ~ propensity

'

Structural_Model

GROUP is a directly observed binary predictor (0,1) indicating control group or treatment group (groups have approximately the same size). All other variables are continuous (7-point Likert scale).

Results of a CFA indicated good fit of the measurement model (χ2/df = 1.44 ; CFI = 0.981 ; RMSEA = 0.039 ; SRMR = 0.036). (GROUP wasn't included in the CFA as it is directly observed).

When I do the structural model including GROUP as exogenous predictor, all of a sudden the SRMR is terribly high:

lavaan 0.6-5 ended normally after 61 iterations

  Estimator                                         ML
  Optimization method                           NLMINB
  Number of free parameters                         64

  Number of observations                           310

Model Test User Model:
                                              Standard      Robust
  Test Statistic                               830.768     798.760
  Degrees of freedom                               370         370
  P-value (Chi-square)                           0.000       0.000
  Scaling correction factor                                  1.040
    for the Satorra-Bentler correction 

Model Test Baseline Model:

  Test statistic                              8462.728    7556.330
  Degrees of freedom                               406         406
  P-value                                        0.000       0.000
  Scaling correction factor                                  1.120

User Model versus Baseline Model:

  Comparative Fit Index (CFI)                    0.943       0.940
  Tucker-Lewis Index (TLI)                       0.937       0.934

  Robust Comparative Fit Index (CFI)                         0.944
  Robust Tucker-Lewis Index (TLI)                            0.939

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)             -12662.869  -12662.869
  Loglikelihood unrestricted model (H1)     -12247.485  -12247.485

  Akaike (AIC)                               25453.738   25453.738
  Bayesian (BIC)                             25692.879   25692.879
  Sample-size adjusted Bayesian (BIC)        25489.895   25489.895

Root Mean Square Error of Approximation:

  RMSEA                                          0.063       0.061
  90 Percent confidence interval - lower         0.058       0.055
  90 Percent confidence interval - upper         0.069       0.067
  P-value RMSEA <= 0.05                          0.000       0.001

  Robust RMSEA                                               0.062
  90 Percent confidence interval - lower                     0.056
  90 Percent confidence interval - upper                     0.068

Standardized Root Mean Square Residual:

  SRMR                                           0.210       0.210

Parameter Estimates:

  Information                                 Expected
  Information saturated (h1) model          Structured
  Standard errors                           Robust.sem

Latent Variables:
                   Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
  propensity =~                                                         
    DTT_04            1.000                               1.463    0.907
    DTT_01            0.854    0.040   21.554    0.000    1.250    0.823
    DTT_02            0.887    0.041   21.635    0.000    1.298    0.840
    DTT_03            0.988    0.042   23.818    0.000    1.447    0.893
  ability =~                                                            
    ABI_02            1.000                               1.540    0.959
    ABI_01            0.915    0.027   34.467    0.000    1.409    0.904
    ABI_03            0.982    0.026   37.687    0.000    1.513    0.940
  benint =~                                                             
    BEN_04            1.000                               1.337    0.846
    BEN_03            0.785    0.068   11.468    0.000    1.050    0.614
    BEN_05            0.847    0.042   20.150    0.000    1.132    0.761
    INT_01            0.764    0.050   15.168    0.000    1.021    0.740
    INT_03            0.937    0.047   19.790    0.000    1.253    0.833
    INT_04            0.818    0.051   16.204    0.000    1.094    0.797
    INT_05            0.917    0.052   17.701    0.000    1.226    0.786
    INT_06            0.983    0.046   21.405    0.000    1.314    0.859
  trust =~                                                              
    TRU_02            1.000                               1.405    0.909
    TRU_03            0.927    0.046   20.333    0.000    1.302    0.895
    TRU_04            0.933    0.052   18.041    0.000    1.311    0.832
  experience =~                                                         
    MOT_X_01          1.000                               1.398    0.836
    MOT_X_02          0.912    0.057   15.962    0.000    1.275    0.853
    MOT_X_03          0.911    0.061   15.048    0.000    1.274    0.848
    MOT_X_04          0.974    0.058   16.749    0.000    1.361    0.848
    MOT_X_05          0.898    0.057   15.804    0.000    1.255    0.814
  money =~                                                              
    MOT_M_01          1.000                               1.719    0.940
    MOT_M_02          0.997    0.024   42.426    0.000    1.715    0.947
    MOT_M_03          0.890    0.035   25.438    0.000    1.531    0.798
    MOT_M_04          0.923    0.029   32.242    0.000    1.587    0.913
    MOT_M_05          0.988    0.027   36.973    0.000    1.699    0.929

Regressions:
                   Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
  experience ~                                                          
    GROUP            -0.107    0.164   -0.651    0.515   -0.077   -0.038
  money ~                                                               
    GROUP             0.436    0.198    2.202    0.028    0.254    0.127
  benint ~                                                              
    experience        0.636    0.052   12.118    0.000    0.665    0.665
    money            -0.132    0.035   -3.785    0.000   -0.170   -0.170
  trust ~                                                               
    ability           0.407    0.046    8.936    0.000    0.447    0.447
    benint            0.347    0.054    6.448    0.000    0.330    0.330
    propensity        0.115    0.051    2.274    0.023    0.120    0.120

Covariances:
                   Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
  propensity ~~                                                         
    ability           0.693    0.136    5.113    0.000    0.308    0.308

Variances:
                   Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
   .DTT_04            0.463    0.065    7.104    0.000    0.463    0.178
   .DTT_01            0.744    0.114    6.545    0.000    0.744    0.322
   .DTT_02            0.701    0.106    6.594    0.000    0.701    0.294
   .DTT_03            0.532    0.081    6.552    0.000    0.532    0.203
   .ABI_02            0.209    0.049    4.272    0.000    0.209    0.081
   .ABI_01            0.442    0.054    8.230    0.000    0.442    0.182
   .ABI_03            0.300    0.062    4.841    0.000    0.300    0.116
   .BEN_04            0.708    0.087    8.116    0.000    0.708    0.284
   .BEN_03            1.818    0.186    9.751    0.000    1.818    0.622
   .BEN_05            0.930    0.096    9.664    0.000    0.930    0.421
   .INT_01            0.860    0.091    9.434    0.000    0.860    0.452
   .INT_03            0.691    0.081    8.576    0.000    0.691    0.306
   .INT_04            0.687    0.089    7.708    0.000    0.687    0.365
   .INT_05            0.929    0.131    7.075    0.000    0.929    0.382
   .INT_06            0.612    0.074    8.244    0.000    0.612    0.262
   .TRU_02            0.412    0.086    4.789    0.000    0.412    0.173
   .TRU_03            0.421    0.083    5.058    0.000    0.421    0.199
   .TRU_04            0.766    0.128    5.975    0.000    0.766    0.308
   .MOT_X_01          0.845    0.111    7.635    0.000    0.845    0.302
   .MOT_X_02          0.611    0.095    6.437    0.000    0.611    0.273
   .MOT_X_03          0.635    0.095    6.717    0.000    0.635    0.281
   .MOT_X_04          0.723    0.124    5.829    0.000    0.723    0.281
   .MOT_X_05          0.805    0.148    5.450    0.000    0.805    0.338
   .MOT_M_01          0.393    0.053    7.363    0.000    0.393    0.117
   .MOT_M_02          0.341    0.057    5.993    0.000    0.341    0.104
   .MOT_M_03          1.334    0.172    7.768    0.000    1.334    0.363
   .MOT_M_04          0.501    0.071    7.102    0.000    0.501    0.166
   .MOT_M_05          0.457    0.081    5.637    0.000    0.457    0.137
    propensity        2.142    0.169   12.702    0.000    1.000    1.000
    ability           2.373    0.171   13.859    0.000    1.000    1.000
   .benint            0.944    0.106    8.915    0.000    0.528    0.528
   .trust             1.271    0.127   10.013    0.000    0.644    0.644
   .experience        1.952    0.201    9.692    0.000    0.999    0.999
   .money             2.908    0.211   13.757    0.000    0.984    0.984

R-Square:
                   Estimate
    DTT_04            0.822
    DTT_01            0.678
    DTT_02            0.706
    DTT_03            0.797
    ABI_02            0.919
    ABI_01            0.818
    ABI_03            0.884
    BEN_04            0.716
    BEN_03            0.378
    BEN_05            0.579
    INT_01            0.548
    INT_03            0.694
    INT_04            0.635
    INT_05            0.618
    INT_06            0.738
    TRU_02            0.827
    TRU_03            0.801
    TRU_04            0.692
    MOT_X_01          0.698
    MOT_X_02          0.727
    MOT_X_03          0.719
    MOT_X_04          0.719
    MOT_X_05          0.662
    MOT_M_01          0.883
    MOT_M_02          0.896
    MOT_M_03          0.637
    MOT_M_04          0.834
    MOT_M_05          0.863
    benint            0.472
    trust             0.356
    experience        0.001
    money             0.016

Because of the high SRMR, I looked at the residuals with resid(sem.fit, type="standardized")$cov. Indeed, many of them are pretty large (.2 up to .6). When I remove GROUP from the structural model SRMR drops dramatically to .075 with none of the residuals even reaching .1.

When I analyse the parts of the structural model separately (Part 1, 2 ,3 as indicated in the diagram), fit is good for the separate parts of the model. So it really seems to be an issue of the overall model.

Questions:

  1. How come residuals "explode" when I include GROUP? Can this be avoided? Could it be because GROUP is binary? Am I missing something important code-wise in that regard?
  2. What can I do to improve model fit? I have already looked at modification indices but none of them make sense from a theoretical point of view. Also, so many residuals are extremely large when including GROUP in the model that I can not remove all of the corresponding variables.
  3. What is the best way to proceed from here? The 𝑅2 of experience and money are pretty low. That would be an indicator of GROUP just not being a very good predictor. Is it justified then to just remove GROUP from the model and say that the original theoretical model just didn't fit?
$\endgroup$
1
$\begingroup$

I do not think that GROUP being binary is itself the issue, but GROUP may be contributing to the worsening of fit between CFA (factor covariances unstructured) and the structural model. GROUP could be contributing in two ways.

First, your model implies that all covariance between Experience and Money is due to those factors' joint dependence on GROUP--and yet, those predictive paths are near 0. This forces Experience and Money to be essentially uncorrelated in the structural model. You could address this by allowing residual covariance between Experience and Money, besides the covariance due to joint dependence on GROUP.

Second, GROUP is only permitted to covary with the other factors through its predictive impact on Experience and Money, and, again, those paths are near 0. You might get a view on this by re-estimating teh model without GROUP at all but with free covariance between Experience and Money. I also might suggest that you re-run the CFA but include GROUP as a single predictor of a "GROUP" factor. Those results will show you whether this variable has strong covariance with any other factor in the model. You could also look at modification indices for the structural model.

Broadly, the worsening of fit means that the constraints on factor covariances implied by the structural model is not entirely consistent with the data.

I might have suggested structuring this as a multiple group model, with the groups defined by GROUP. That would get binary GROUP out of your hair, statistically. But your sample size is small for that. Still, you might well impose measurement invariance across the two subsamples, and that would sharply cut the number of additional parameters you would be estimating. You could still capture the impact of GROUP on Experience and Money by looking at differences in their factor means.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your answer. I have already run CFA with GROUP as LV. Its covariance with other LVs is almost non-existent: 0.016 0.002 -0.057 -0.045 -0.024 0.109 0.250. So that means, that group simply does not covary with any other variable and is therefore not a suitable predictor? $\endgroup$ – Felix Krauth Jan 18 at 16:39
  • $\begingroup$ Whoa, whoa. If these are covariances, it is hard to tell what is small. Those last two numbers may well be not small. GROUP might also serve as a moderator--something that you might see in a multi-group analysis. I am assuming that GROUP is fairly balanced, with about half of respondents in each condition. Lack of balance could contribute to low predictive power. Also, I should have added that yes, SRMR seems disproportionate to the values of the other fit indices. $\endgroup$ – Ed Rigdon Jan 18 at 16:44
  • 1
    $\begingroup$ Indeed GROUP is split 50/50. I will try a multi-group model and see where that gets me. Thank you for the suggestion. $\endgroup$ – Felix Krauth Jan 18 at 16:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.