I'm in the process of fitting measurement invariance models using the semTools
package for R. For the basic measurement model--not distinguishing between groups--I used a robust ML estimator (MLR
), and everything seemed fine. But I noticed that when I used this robust estimator (or an alternative robust estimator like MLM
) when testing measurement invariance, my models yielded fit indexes that seemed highly implausible, if not impossible. Specifically, the CFI
and the RMSEA
both improve after constraining loadings and intercepts to equality, rather than worsening (as they ought to). When I re-run these models with a traditional ML estimator, however, this problem disappears (but then I do not get the benefits of correcting my test statistic for non-normality in my indicators).
Any guidance or ideas about what may be going on here would be seriously appreciated.
Update 1
Okay, as requested by Jeremy Miles, here's an update with some model code and output, in order to better diagnose the issue:
Update 2
Posted some factor loadings and latent covariances from both groups, to help diagnose why the scaling factor is changing so much between models.
Model Syntax
library(semTools)
library (lavaan)
RNS.invar<-"
manage =~ RNS_3 + RNS_8 + RNS_6 + RNS_19 + RNS_13 + RNS_4 + RNS_12 + RNS_15 + RNS_1 + RNS_22 + RNS_18 + RNS_5 + RNS_24
agree =~ RNS_8 + RNS_19 + RNS_13 + RNS_12 + RNS_15 + RNS_1 + RNS_18 + RNS_17 + RNS_11 + RNS_14 + RNS_7 + RNS_9 + RNS_16 + RNS_5 + RNS_2
explicit =~ RNS_3 + RNS_20 + RNS_4 + RNS_1 + RNS_22 + RNS_18 + RNS_17 + RNS_9 + RNS_16 + RNS_10 + RNS_5 + RNS_2
punish =~ RNS_12 + RNS_22 + RNS_9 + RNS_5 + RNS_21 + RNS_20 + RNS_23 + RNS_24
"
invariance=measurementInvariance(RNS.invar, data=dat, group="Male", estimator="MLR")
Output: invariance$fit.configural
Model Fit Indexes
Estimator ML Robust
Minimum Function Test Statistic 983.743 890.242
Degrees of freedom 444 444
P-value (Chi-square) 0.000 0.000
Scaling correction factor 1.105
for the Yuan-Bentler correction
Comparative Fit Index (CFI) 0.858 0.846
RMSEA 0.100 0.091
Factor Loadings
Group 1 [1]:
Latent Variables:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage =~
RNS_3 1.000 1.428 0.865
RNS_8 0.813 0.088 9.201 0.000 1.161 0.673
RNS_6 0.988 0.088 11.189 0.000 1.411 0.811
RNS_19 0.889 0.116 7.680 0.000 1.270 0.694
Group 2 [2]:
Latent Variables:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage =~
RNS_3 1.000 1.549 0.907
RNS_8 0.798 0.091 8.741 0.000 1.237 0.823
RNS_6 0.982 0.102 9.617 0.000 1.521 0.861
RNS_19 0.728 0.121 6.002 0.000 1.127 0.695
Latent Covariances
Group 1 [1]:
Covariances:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage ~~
agree 0.190 0.098 1.948 0.051 0.382 0.382
explicit 0.004 0.020 0.183 0.855 0.174 0.174
punish 0.056 0.051 1.095 0.274 0.324 0.324
agree ~~
explicit -0.002 0.012 -0.177 0.859 -0.400 -0.400
punish 0.001 0.005 0.171 0.865 0.021 0.021
explicit ~~
punish -0.000 0.001 -0.152 0.879 -0.085 -0.085
Group 2 [2]:
Covariances:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage ~~
agree 0.063 0.066 0.963 0.336 0.137 0.137
explicit 0.039 0.167 0.232 0.817 0.877 0.877
punish 0.015 0.257 0.059 0.953 0.415 0.415
agree ~~
explicit 0.000 0.003 0.184 0.854 0.057 0.057
punish -0.001 0.023 -0.059 0.953 -0.195 -0.195
explicit ~~
punish 0.000 0.006 0.057 0.955 0.519 0.519
Output: invariance$fit.loadings
Model Fit Indexes
Estimator ML Robust
Minimum Function Test Statistic 1021.659 861.367
Degrees of freedom 488 488
P-value (Chi-square) 0.000 0.000
Scaling correction factor 1.186
for the Yuan-Bentler correction
Comparative Fit Index (CFI) 0.859 0.871
RMSEA 0.095 0.079
Factor Loadings
Group 1 [1]:
Latent Variables:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage =~
RNS_3 1.000 1.462 0.873
RNS_8 (.p2.) 0.791 0.054 14.673 0.000 1.156 0.678
RNS_6 (.p3.) 0.981 0.053 18.469 0.000 1.435 0.817
RNS_19 (.p4.) 0.777 0.079 9.807 0.000 1.136 0.644
Group 2 [2]:
Latent Variables:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage =~
RNS_3 1.000 1.536 0.913
RNS_8 (.p2.) 0.791 0.054 14.673 0.000 1.215 0.800
RNS_6 (.p3.) 0.981 0.053 18.469 0.000 1.508 0.863
RNS_19 (.p4.) 0.777 0.079 9.807 0.000 1.194 0.698
Latent Covariances
Group 1 [1]:
Covariances:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage ~~
agree 0.177 0.049 3.592 0.000 0.392 0.392
explicit 0.001 0.001 1.683 0.092 0.164 0.164
punish 0.005 0.002 2.877 0.004 0.304 0.304
agree ~~
explicit -0.001 0.000 -3.217 0.001 -0.389 -0.389
punish 0.000 0.000 0.000 1.000 0.000 0.000
explicit ~~
punish -0.000 0.000 -0.479 0.632 -0.061 -0.061
Group 2 [2]:
Covariances:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage ~~
agree 0.113 0.063 1.798 0.072 0.221 0.221
explicit 0.004 0.001 2.745 0.006 0.335 0.335
punish 0.010 0.002 3.948 0.000 0.455 0.455
agree ~~
explicit -0.001 0.000 -4.246 0.000 -0.527 -0.527
punish -0.000 0.001 -0.476 0.634 -0.060 -0.060
explicit ~~
punish 0.000 0.000 1.130 0.259 0.149 0.149
Output: invariance$fit.intercepts
Model Fit Indexes
Estimator ML Robust
Minimum Function Test Statistic 1049.546 754.119
Degrees of freedom 508 508
P-value (Chi-square) 0.000 0.000
Scaling correction factor 1.392
for the Yuan-Bentler correction
Comparative Fit Index (CFI) 0.857 0.915
RMSEA 0.094 0.063
Factor Loadings
Group 1 [1]:
Latent Variables:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage =~
RNS_3 1.000 1.460 0.872
RNS_8 (.p2.) 0.802 0.047 17.113 0.000 1.171 0.678
RNS_6 (.p3.) 0.986 0.044 22.531 0.000 1.440 0.817
RNS_19 (.p4.) 0.786 0.054 14.597 0.000 1.148 0.648
Group 2 [2]:
Latent Variables:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage =~
RNS_3 1.000 1.532 0.913
RNS_8 (.p2.) 0.802 0.047 17.157 0.000 1.229 0.800
RNS_6 (.p3.) 0.986 0.044 22.278 0.000 1.512 0.863
RNS_19 (.p4.) 0.786 0.054 14.504 0.000 1.205 0.700
Latent Covariances
Group 1 [1]:
Covariances:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage ~~
agree 0.175 0.044 3.964 0.000 0.391 0.391
explicit 0.001 0.001 1.711 0.087 0.167 0.167
punish 0.004 0.001 3.116 0.002 0.306 0.306
agree ~~
explicit -0.001 0.000 -5.071 0.000 -0.391 -0.391
punish 0.000 0.000 0.003 0.997 0.000 0.000
explicit ~~
punish -0.000 0.000 -0.338 0.735 -0.059 -0.059
Group 2 [2]:
Covariances:
Estimate Std.Err Z-value P(>|z|) Std.lv Std.all
manage ~~
agree 0.111 0.054 2.045 0.041 0.220 0.220
explicit 0.003 0.001 3.713 0.000 0.340 0.340
punish 0.007 0.001 5.103 0.000 0.454 0.454
agree ~~
explicit -0.001 0.000 -6.576 0.000 -0.530 -0.530
punish -0.000 0.000 -0.520 0.603 -0.062 -0.062
explicit ~~
punish 0.000 0.000 1.139 0.255 0.153 0.153
Summary
As I would expect, the $\chi^2$ statistic increases with each added level of invariance, when estimated using the boring ol' ML estimator. But the robust estimates of the $\chi^2$ statistic are decreasing with each added level of invariance constraints, when estimated using the MLR estimator...