Comparing Individuals Scores on Different Latent Factors

Question

I am looking to compare individual's scores on different factors which have been computed based on responses to a survey. Specifically, the survey data corresponds to 4 factors, [call them Factor A, Factor B, Factor C, and Factor D] and I would like to know whether any individual, or group of individuals, has a propensity for Factor A over Factor B (or other pairwise comparisons). Each computed factor is scored out of 5, with the higher scores representing a propensity for that factor. The factors are "distinct" and "independent"

If I am overlooking what should be a simple statistical test, please do let me know! Otherwise, I will go into slightly more detail below. I would not put it past myself to be thinking too much into this, or overlooking the obvious!

---

Survey data was collected which was based on previous research that derived a four-factor scale (think similar to MBTI or Big-5). A confirmatory factor analysis was run on the collected data to ensure that the factor structure made sense, and the results appear to be significant. With this data there are two types of comparisons that seem sensible to make:

1. Compare how different groups score relative to one another on the different factors. If I am not mistaken, this is relatively straightforward testing (i.e. using ANOVA + post-hoc analysis or t-testing or similar). The previous literature focused on these inter-group comparisons (men vs. women, etc.), and mostly studied two-distinct groups with simple T-testing.

2. Compare how the 4 factors vary within any given group, or by the individual. That is: if respondent A scores a 5 on Factor A and a 3 on Factor B, can we say anything statistically about this difference? Is there a test that can talk about relative affinities for the different factors given some group correspondence?

So an example research question would be: "Do women have a stronger affinity for Factor A as compared to Factor B?"

Part of me is skeptical that such a test could exist, and that it would take more of a theoretical development of the underlying factors to ensure that they can be compared apples-to-apples.

In theory, the question "Does individual A really score higher on A versus B?" is a valid question that a researcher may want to know, I do not know that the scale by which this is being measured makes sense to compare?

I apologize if I am missing something very obvious, and I thank you for any resources you can provide for me. Let me know if something needs to be clarified!

Answer 1

Question 1.

This is totally doable, and ANOVAs and t-tests are one way that you could go about comparing groups on their scores. Typically, this would involve you calculating some average or sum score for each individual to represent their standing on a given factor. Given that you have already run a CFA on all the factors, however, this approach would miss several opportunities for more rigorous analyses.

The t-testing/ANOVA approach above makes two particularly relevant assumptions:

1. When you calculate a sum or average score, you assume that each item within a given factor is equally important for the individual's factor score--that all items are equally representative of the factor
2. You assume that the underlying factor(s) you are comparing are the same between groups--not that they are at the same level, but that you are effectively comparing apples to apples, so to speak.

If you look at the results of the CFA you already ran, it is likely that you can already rule out the tenability of the first assumption: different factor loading values across items imply that some items are stronger indicators of the factor than others (you could perform a statistical test to evaluate this assumption formally, though this is often not done).

The second assumption is one that you can--and in my opinion should--formally test, especially given that you already have the CFA analysis in hand: this is how you evaluate measurement invariance between groups (see Vandenberg & Lance, 2000; Little, 2013, or Beaujean, 2014 for accessible overviews). If you were to compare latent means of groups and find significant differences, it might be because the groups actually have different average levels of the latent factor, but it also might be because you're inadvertently assessing different factors in each group--this is what testing measurement invariance allows you to rule out.

If all checks out (and to validly compare latent means, you want to establish configural, weak/metric, and strong/intercept invariance) you can actually do a latent version of an ANOVA (or a t-test) in SEM, which is a more accurate and statistically powerful test of those comparisons. This process, in its totality, is too complicated to walk through step-by-step here, but you might find my answer on this thread to be helpful (and I refer to the same resources that I've cited here, among others). But in a nutshell, you are in a position to perform more rigorous analyses that make fewer assumptions about the measurement of your factors, so you might want to take advantage of what a latent variable modelling approach could afford for testing your questions.

Question 2

It sounds like your second question could mean testing two different things. With the first, you could compare mean levels of all factors across participants (or within a group) if you consider factor a within-subject factor, which would require you to model the dependency between indicators of your factors within-person. Otherwise the process could proceed in the same way that I've described above: establishing measurement invariance, and then proceeding with a latent ANOVA (or t-test).

As for the second possibility--"compare how the 4 factors vary within any individual..." will likely be much more complicated. I'm not terrifically knowledgable about these kinds of analyses, but if this is what you want, it sounds like you may want to look into the literature on fitting intensive individual longitudinal models.

References

Beaujean, A. A. (2014). Latent variable modeling using R: A step-by-step guide. New York, NY: Routledge.

Little, T. D. (2013). Longitudinal structural equation modeling. New York, NY: Guilford Press.

Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational researchers. Organizational Research Methods, 3, 4-70.