# In measurement invariance testing, how can one differentiate between differences in response styles and differences in latent means?

I am trying to establish measurement invariance between two groups on a depression measure: high-school aged boys and high-school aged girls. Though my entire sample reports elevated depression symptoms, girls report higher levels of symptoms than boys.

In measurement invariance testing, "scalar invariance" (full score equivalence) is supported when both the factor loadings and the factor intercepts are equivalent across groups. This means that respondents in both groups use the measure the same; they will both select the same response option given the same level of latent depression. When you look at respondents' response functions, the slopes and intercepts are the same (source):

This does not hold for my analysis. Because girls report more depression, their factor intercepts are higher. Thus, this measure fails the measurement invariance test.

Here is my question: (How) is it possible to differentiate between differences in respondents' response styles (e.g., girls tend to over-report depression) and differences in the actual latent mean (e.g., girls are more depressed)? When I started this project, I assumed the point of measurement invariance testing would be to figure this out. But now I'm wondering if this is even possible.

## What are we testing in measurement invariance?

The first theorem of Meredith (1993) (p. 528) states that a random variable $$X$$ is measurement invariant with regards to selection on $$V$$ if and only if they are locally independent when conditioned on $$\eta$$, for every $$\eta$$ in the sample space.

Consider now that $$X$$ are your depression scale items, $$\eta$$ is your latent depression construct score and $$V$$ is sex. What measurement invariance implies is that, in the diagram $$V \rightarrow \eta \rightarrow X$$, conditioning on the mediator $$\eta$$ should block any effect that $$V$$ could have on $$X$$, i.e. there is no direct effect from $$V$$ to $$X$$. The only way that sex ($$V$$) can influence depression item scores ($$X$$) is through a difference in the latent construct score ($$\eta$$).

In scalar invariance (or strong factorial invariance), both the slopes of the effect of the latent common scores on the item scores and the item intercepts are assumed equal across groups. Under this assumption, if girls report higher average scores in the observed items $$X$$, it must be due to higher average latent depression scores $$\eta$$.

Note that you incorrectly state

This does not hold for my analysis. Because girls report more depression, their factor intercepts are higher. Thus, this measure fails the measurement invariance test.

Girls having higher latent factor intercepts does not violate measurement invariance nor scalar invariance. As we saw in the diagram, there is no problem with sex influencing directly on the latent factor score ($$V \rightarrow \eta$$). The issue is when sex would influence the item scores directly without reflecting a change in the latent factor ($$V \rightarrow X$$).

Then, your question is no longer a statistical one, but at least a psychological one if not an epistemological one. What you would like to know is whether the average increase in item scores for girls is reflecting an increase in actual latent depression score, or just a difference in social desirability for reporting in contrast to boys. Also note that you say that girls over-report depression, which already reflects some unwarranted implicit assumptions (i.e. it is not boys that under-report depression).

Without further assumptions which would strongly depend on domain knowledge (that might not even exist), you cannot turn this into a testable statistical problem. For instance, what specific functional form would you expect for the effect of desirability on reporting? Should it affect all sub-scales or items in the same way? Would only a subgroup of girls be affected? etc.

• Thank you for your response! This is a good answer to the question which, as you point out, has some erroneous assumptions.
– ila
Commented Apr 24, 2023 at 19:15

To clarify, measurement (or metric) invariance means the "respondents in both groups use the measure the same." However, scalar invariance is used to test if the means of the latent variable are different between groups. To clarify, this does NOT mean that respondents won't give the same response option for the same level of latent depression. Scalar invariance instead means that the average for the latent variables for the groups being measured are actually different (assuming you've established metric invariance).

If you wish to tackle the question of response styles (patterns of one group responding to Likert-type questions in a different fashion from another), this is another level of analysis for latent variable modeling. And, if you wish to show that your dependent variable construct is the "same" after you account for response styles, this will require having a latent grouping variable to indicate which type of response style is present. And then you will need to confirm (or not) that this response style latent variable "intercept" is different for your different groups.

I will forewarn you that this type of analysis requires a rather large number of respondents in the data set to be effective (e.g., to have enough power to detect both the latent construct of interest and the latent construct representing the response style).

Happy to share more if you're interested.

• Metric invariance does not mean that respondents in both groups use the measure the same. Members of one group can systematically over- or under-report relative to the other group and you can still find metric invariance. I also think scalar invariance also does not refer to equivalence or non-equivalence of latent means between groups, that is a different test. See journals.sagepub.com/doi/abs/10.1177/109442810031002
– ila
Commented Apr 16, 2023 at 1:41
• Sorry to disagree, but metric (measurement) invariance does indeed mean that the measurement tool is indeed measuring the same thing. Please see the following encyclopedia entry for multiple group SEM: sk.sagepub.com/reference/… Commented Apr 16, 2023 at 13:09