I am running a measurement invariance in lavaan in R. I am comparing a measure of sociocultural appearance pressures across White and Black women. Because this is a 5-point scale, I am treating the items as ordinal. The presented x2 and fit indices were from the robust (WLSMV) estimator.
I have large, but uneven samples. There are 553 White and 273 Black women. I'm evaluating a combo of change in x2 and fit indices. The fit indices are generally adequate for the initial cfa: RMSEA=.085, 95% CI [.081, .090], TLI=.983; SRMR=.054, χ2(199)=1346.800, p<.001.
The configural model is somewhat similar: RMSEA=.088, 95% CI [.083, .092]; TLI=.982; SRMR=.061, χ2(398)=1612.698, p<.001.
The metric model had the following changes: ΔRMSEA=-.009, ΔTLI=.001, and ΔSRMR=.002. Only the SRMR suggested a decrement in model fit, but the change (.002) was below the recommended cut-off of .025 by Chen (2007). The adjusted chi-square difference test was significant, Δχ2(22)=38.62308, p=.016 (I used the scaling correction factors, etc. to calculate the change in the robust x2 estimates).
This is a tricky one for me. I see so much controversy over using the change in chi-square. The fit indices changes were so minimal. My co-author wants us to ignore the chi-square.
I start to question my data when I run the scalar model. In the scalar model, I constrained loadings and thresholds to equality. With this model, I was surprised to see that chi-square goes down, from 1620.556 (420 df) in the metric model to 1462.593 (481 df) in the scalar model. When I run the x2 difference test, I get a p value of 1 because the chi-square went down instead of up. Again, the fit indices changes are minimal, compared to the metric model: ΔRMSEA=-.002, ΔTLI=.003, and ΔSRMR=-.001.
So, I guess my questions are: do you all agree that I should ignore the significant change in x2 from the configural to metric model? And is it OK for the chi-square to decrease from the metric model to the scalar model?