When is it acceptable to collapse across groups when performing a factor analysis?

Question

Imagine that

• responses were collected on a 20 item scale which was designed to measure 4 factors with 5 items on each scale.
• participants were drawn from two groups (Group 1) and (Group 2) with sample size $n_1 = 150$ and $n_2 = 150$.
• a researcher wanted to assess the factor structure of the scale

Common scenarios that I see in my consulting:

• Group 1 are first year psychology students and Group 2 is sampled from the general community
• Group 1 are sampled at one period of time and Group 2 is sampled several years later
• Group 1 is a normal population and Group 2 is a clinical population

Question

1. Under what circumstances would it be appropriate to collapse across groups?
2. How would these circumstances be assessed?

My initial Thoughts:

My own initial thoughts were as follows:

1. Theoretical assessment: assess the degree to which the two groups were sampled or measured in ways that would alter the means, sds, or correlation between the items
2. Empirical assessment: Examine differences between means, sds, and intercorrelations on the scales and optionally on other relevant variables (e.g., demographics); perform a two-group confirmatory factor analysis to assess the consistency of the factor structure across groups.

Essentially, if the empirical evidence suggests that the groups are similar and the theoretical assessment suggests that they are similar, then it should be reasonable to combine.

Conclusion

• Does the approach above seem reasonable?
• Do you have alternative strategies?
• Are there any references that provide recommendations or examples regarding best practice in this situation?

Answer 1

There seems to be two cases to consider, depending on whether your scale was already validated using standard psychometric methods (from classical test or item response theory). In what follows, I will consider the first case where I assume preliminary studies have demonstrated construct validity and scores reliability for your scale.

In this case, there is no formal need to apply exploratory factor analysis, unless you want to examine the pattern matrix within each group (but I generally do it, just to ensure that there are no items that unexpectedly highlight low factor loading or cross-load onto different factors); in order to be able to pool all your data, you need to use a multi-group factor analysis (hence, a confirmatory approach as you suggest), which basically amount to add extra parameters for testing a group effect on factor loading (1st order model) or factor correlation (2nd order model, if this makes sense) which would impact measurement invariance across subgroups of respondents. This can be done using Mplus (see the discussion about CFA there) or Mx (e.g. Conor et al., 2009), not sure about Amos as it seems to be restricted to simple factor structure. The Mx software has been redesigned to work within the R environment, OpenMx. The wiki is well responding so you can ask questions if you encounter difficulties with it. There is also a more recent package, lavaan, which appears to be a promising package for SEMs.

Alternatives models coming from IRT may also be considered, including a Latent Regression Rasch Model (for each scale separately, see De Boeck and Wilson, 2004), or a Multivariate Mixture Rasch Model (von Davier and Carstensen, 2007). You can take a look at Volume 20 of the Journal of Statistical Software, entirely devoted to psychometrics in R, for further information about IRT modeling with R. You may be able to reach similar tests using Structural Equation Modeling, though.

If factor structure proves to be equivalent across the two groups, then you can aggregate the scores (on your four summated scales) and report your statistics as usual. However, it is always a challenging task to use CFA since not rejecting H0 does by no mean allow you to check that your postulated theoretical model is correct in the true world, but just that there is no reason to reject it on statistical grounds; on the other hand, rejecting the null would lead to accept the alternative, which is generally left unspecified, unless you apply sequential testing of nested models. Anyway, this is the way we go in cross-cultural settings, especially when we want to assess whether a given questionnaire (e.g., on Patients Reported Outcomes) measures what it purports to do whatever the population it is administered to.

Now, regarding the apparent differences between the two groups -- one is drawn from a population of students, the other is a clinical sample, assessed at a later date -- it depends very much on your own considerations: Does mixing of these two samples makes sense from the literature surrounding the questionnaire used (esp., it should have shown temporal stability and applicability in a wide population), do you plan to generalize your findings over a larger population (obviously, you gain power by increasing sample size). At first sight, I would say that you need to ensure that both groups are comparable with respect to the characteristics thought to influence one's score on this questionnaire (e.g., gender, age, SES, biomedical history, etc.), and this can be done using classical statistics for two-groups comparison (on raw scores). It is worth noting that in clinical studies, we face the reverse situation: We usually want to show that scores differ between different clinical subgroups (or between treated and naive patients), which is often refered to as know-group validity.

Reference:

1. De Boeck, P. and Wilson, M. (2004). Explanatory Item Response Models. A Generalized Linear and Nonlinear Approach. Springer.
2. von Davier, M. and Carstensen, C.H. (2007). Multivariate and Mixture Distribution Rasch Models. Springer.

Answer 2

It might be a little fly by night, but your theory may suggest whether the two groups have the same factor structure or not. If your theory suggests they do, and there is no reason to doubt the theory, I'd suggest you could go right ahead and trust that they have the same factor structure.

Your empirical assessment would probably be a good route to go just to spot check the theoretical assessment as whether they are likely to share the same structure. However, I don't intuitively see why mean differences between items would imply they have a different underlying factor structure. It seems to me that might just suggest that one group has higher or lower scores on a given factor.

Answer 3

The approach you mention seems reasonable, but you'd have to take into account that you cannot see the total dataset as a single population. So theoretically, you should use any kind of method that can take differences between those groups into account, similar to using "group" as a random term in an ANOVA or GLM approach.

An alternative for empirical evaluation would be to check formally whether an effect of group can be found on the answers. To do that, you could create a binary dataset with following columns :

yes/no - item - participant - group

With this you can use item as a random term, participant nested in group and test the fixed effect of group using e.g. a glm with a logit link. You can just ignore participant too if you lose too many df.

This is an approximation of the truth, but if the effect of group is significant, I wouldn't collapse the dataset.