Best practice for establishing configural invariance? I was hoping for some perspectives on the best practices for establishing configural invariance using ordered indicators with WLSMV estimation. I have a 3-factor scale and I am assessing invariance across two groups. I have some question about whether the factor structure is truly invariant across groups. I know that typically when assessing configural invariance, you are just looking at overall model fit, because there is no model to compare it to. However, I have seen some researchers assess model fit in each group as a primary step.
When assessing configural invariance overall, fit indices seem generally good, though RMSEA is slightly high:
CFI: .993
TLI: .986
RMSEA: .083
SRMR: .028
However, I also subset the data and ran the CFA in each group separately. Here are fit statistics:
Group 1 (n=199)
CFI: .994
TLI: .989
RMSEA: .066
SRMR: .026
Group 2 (n=368)
CFI: .992
TLI: .984
RMSEA: .091
SRMR: .029
I notice that the RMSEA is considerably higher in Group 2, which is also a bigger sample size. Should I be concerned about the difference in RMSEA? When looking at modification indices, it seems like in Group 1, one of the items might load better on a different factor. The MI isn't super high (10ish), but it's almost 0 in the other group.
Is this concerning enough to consider that the factor structure of this measure is not invariant? Should I have even looked at fit within each group? Thanks for any thoughts! It seems like lack of configural invariance is relatively rare, and I am struggling to find many resources on how to assess it beyond just looking at overall model fit.
 A: I think it's a good idea to look at fit within each group. Configural invariance is (relatively) rare, but it can be interesting.
If often occurs because of interpretation differences. For example, Saucier's big 5  mini-markers scale contains the word 'touchy', to American students, this loads on neuroticism, when I gave it to British students, it loads on agreeableness, because British students interpret it to mean 'tactile'. (Similarly, the word 'fussy' to a British person means 'selective', not concerned about detail.
Health outcome scales in the US often include items about affording health care. To people in other countries, this question is confusing. Similarly, outcome scales can contain items about sexual activity, for the very old and the young, these are irrelevant.
Time can also have an effect. The original Eysenck Personality Inventory, from the 1960s, contains the item "Can you usually let yourself go and enjoy yourself a lot at a gay party?” That means something rather different nowadays.
tl;dr: Think about this as a theoretical / conceptual problem, not a statistics problem.
