Should residuals be equal (or unequal) in multigroup SEM when testing means?

Question

What happens if I allow residual variances (and their correlations) to be freely estimated when testing multigroup differences in intercepts?

I am comparing intercepts of my dependent variables (e.g., Dep1, Dep2..) between (latent) groups (Group1 and Group2) in structural equation framework (using Mplus), with couple of covariates (covariate1, covariate2..). By default the residuals regarding the dependent variables are constrained to be equal between the groups. Yet, when I allow the residuals (as well as their covariances) to be freely estimated within each group, there is considerable change in my results regarding differences in the intercepts.

When using latent factors this issue is often discussed as a final test, and sometimes unnecessary, when testing factorial invariance and measurement invariance (e.g., "strict or invariant uniqueness"). However, in my case, the dependent variables are not latent, but are simple observed variables.

I am puzzled about which model should I trust. As the model fit (BIC) seems to increase, then perhaps the model with freely estimated residuals? Standard errors of the intercepts seem to increase slightly. I don't understand what causes the mean tests to differ here: When the residuals are free, I obtain more significant results. Perhaps the covariates have different effects on the dependent variables (i.e., moderated by the groups)? Or is some common shared variance between the dependent variables modeled more efficiently as error variance?

Pseudocode as follows:

Overall model:  !will be estimated similarly for both Group1 and Group2
Dep1 ON covariate1 covariate2;

Model for Group1:
[Dep1] (a1); ! estimate intercept
[Dep2] (b2); ! estimate intercept

Model for Group2:
[Dep1] (a2); ! estimate intercept
[Dep2] (b2); ! estimate intercept

--> And in this model I test the difference between "a1 and a2", and between "b1 and b2". By default Mplus assumes equal residuals of Dep1 and Dep2 between the groups (residuals, as there are two covariates involved). But I can estimate them freely (and their correlations) by adding the following lines to both Group1 and Group2:

(Dep1); ! estimate variance/residuals freely for this group
(Dep2); ! estimate variance/residuals freely for this group

Dep1 WITH Dep2; !estimate the covariance freely for this group

Answer 1

Indeed, I would recommend allowing for heteroskedasticity across groups by freeing residual variances. Constraining them to equality will force the slopes to adjust accordingly, so that the total variance of the variables is reproduced as closely as possible (in balance with covariances, of course). Is the BIC lower (better) when freeing residuals across groups? Even if it gets bigger, you might want to prioritize having unbiased estimates of the slopes and intercepts. The SEs will increase a bit for each additional free parameter, because you are using more of your observed information to estimate more parameters. But if there truly is heteroskedasticity in the population, then it should be worth trading precision to reduce bias.