Let's say I have a dataset with scores on a bunch of questionnaire items, which are theoretically comprised of a smaller number of scales, like in psychology research.
I know a common approach here is to check the reliability of the scales using Cronbach's alpha or something similar, then aggregate the items in the scales to form scale scores and continue analysis from there.
But there's also factor analysis, which can take all of your item scores as input and tell you which of them form consistent factors. You can get a sense of how strong these factors are by looking at loadings and communalities, and so on. To me this sounds like the same kind of thing, only much more in-depth.
Even if all your scale reliabilities are good, an EFA might correct you on which items fit better into which scales, right? You're probably going to get cross loadings and it might make more sense to use derived factor scores than simple scale sums.
If I want to use these scales for some later analysis (like regression or ANOVA), should I just aggregate the scales so long as their reliability holds up? Or is something like CFA (testing to see if the scales hold up as good factors, which seems to be measuring the same thing as 'reliability').
I've been taught about both approaches independently and so I really don't know how they relate, whether they can be used together or which one makes more sense for which context. Is there a decision tree for good research practice in this case? Something like:
Run CFA according to predicted scale items
- If CFA shows good fit, calculate factor scores and use those for analysis.
- If CFA shows poor fit, run EFA instead and take exploratory approach (or something).
Are factor analysis and reliability testing indeed separate approaches to the same thing, or am I misunderstanding somewhere?
