I am a newbie here. I searched through the forum but did not find similar questions, so I hope someone who is proficient with SEM / lavaan can help out.
I am using R lavaan and semTools to perform measurement invariance analysis based on the following models:
fit.original <- lavaan::cfa(model = cfa_model_expr, data = data_df, ordered = TRUE, group = "GENDER")
fit.metric <- lavaan::cfa(model = cfa_model_expr, data = data_df, ordered = TRUE, group = "GENDER", group.equal = c("loadings"))
fit.scalar <- lavaan::cfa(model = cfa_model_expr, data = data_df, ordered = TRUE, group = "GENDER", group.equal = c("loadings", "intercepts"))
Once fitted, I used semTools::compareFit to compare these three models, but I got lavaan warning that some models were not nested or less restricted model.
Warning: lavaan WARNING:
Some restricted models fit better than less restricted models;
either these models are not nested, or the less restricted model
failed to reach a global optimum. Smallest difference =
-540.550720801662
Scaled Chi-Squared Difference Test (method = “satorra.2000”)
lavaan NOTE:
The “Chisq” column contains standard test statistics, not the
robust test that should be reported per model. A robust difference
test is a function of two standard (not robust) statistics.
Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)
m.original 1678 3786.0
m.metric 1714 4463.5 102.06 36 0.00000003094 ***
m.scalar 1836 3923.0 -717.11 122 1
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
So I proceed to use anova to do pairwise comparison:
Scaled Chi-Squared Difference Test (method = “satorra.2000”)
lavaan NOTE:
The “Chisq” column contains standard test statistics, not the
robust test that should be reported per model. A robust difference
test is a function of two standard (not robust) statistics.
Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)
m.original 1678 3786.0
m.metric 1714 4463.5 102.06 36 0.00000003094 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Warning: lavaan WARNING:
Some restricted models fit better than less restricted models;
either these models are not nested, or the less restricted model
failed to reach a global optimum. Smallest difference =
-540.550720801662
Scaled Chi-Squared Difference Test (method = “satorra.2000”)
lavaan NOTE:
The “Chisq” column contains standard test statistics, not the
robust test that should be reported per model. A robust difference
test is a function of two standard (not robust) statistics.
Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)
m.metric 1714 4463.5
m.scalar 1836 3923.0 -717.11 122 1
So the culprit appears to be between m.metric and m.scalar. Then if I compare m.original and m.scalar, I got a non-significant result:
Scaled Chi-Squared Difference Test (method = “satorra.2000”)
lavaan NOTE:
The “Chisq” column contains standard test statistics, not the
robust test that should be reported per model. A robust difference
test is a function of two standard (not robust) statistics.
Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)
m.original 1678 3786
m.scalar 1836 3923 185.4 158 0.06715 .
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
It does not make sense here because that means my original model satisfied the factor loadings + intercepts invariance between groups but not factor loadings itself (weak invariance)?
UPDATE (2 May 2024): To include the summary statistics for fit.scalar below. The model converged and the model fit statistics seem to suggest an adequate fit (with scaled CFI ~ 0.96, scaled TLI ~ 0.96, scaled RMSEA ~0.065, SRMR ~ 0.055). Could this be a case of sample size issues for chi-square (I have N=699 in total, divided into 345 and 354 for group 1 and group 2)
lavaan 0.6.16 ended normally after 201 iterations
Estimator DWLS
Optimization method NLMINB
Number of model parameters 522
Number of equality constraints 208
Number of observations per group:
Male 345
Female 354
Model Test User Model:
Standard Scaled
Test Statistic 3922.985 4517.138
Degrees of freedom 1836 1836
P-value (Chi-square) 0.000 0.000
Scaling correction factor 1.181
Shift parameter 1196.628
simple second-order correction
Test statistic for each group:
Male 2181.037 2436.694
Female 1741.948 2080.445
Model Test Baseline Model:
Test statistic 516568.414 65863.161
Degrees of freedom 1806 1806
P-value 0.000 0.000
Scaling correction factor 8.036
User Model versus Baseline Model:
Comparative Fit Index (CFI) 0.996 0.958
Tucker-Lewis Index (TLI) 0.996 0.959
Robust Comparative Fit Index (CFI) NA
Robust Tucker-Lewis Index (TLI) NA
Root Mean Square Error of Approximation:
RMSEA 0.057 0.065
90 Percent confidence interval - lower 0.055 0.062
90 Percent confidence interval - upper 0.060 0.067
P-value H_0: RMSEA <= 0.050 0.000 0.000
P-value H_0: RMSEA >= 0.080 0.000 0.000
Robust RMSEA NA
90 Percent confidence interval - lower NA
90 Percent confidence interval - upper NA
P-value H_0: Robust RMSEA <= 0.050 NA
P-value H_0: Robust RMSEA >= 0.080 NA
Standardized Root Mean Square Residual:
SRMR 0.055 0.055
Parameter Estimates:
Standard errors Robust.sem
Information Expected
Information saturated (h1) model Unstructured
