9
$\begingroup$

Let's say I have a dataset with scores on a bunch of questionnaire items, which are theoretically comprised of a smaller number of scales, like in psychology research.

I know a common approach here is to check the reliability of the scales using Cronbach's alpha or something similar, then aggregate the items in the scales to form scale scores and continue analysis from there.

But there's also factor analysis, which can take all of your item scores as input and tell you which of them form consistent factors. You can get a sense of how strong these factors are by looking at loadings and communalities, and so on. To me this sounds like the same kind of thing, only much more in-depth.

Even if all your scale reliabilities are good, an EFA might correct you on which items fit better into which scales, right? You're probably going to get cross loadings and it might make more sense to use derived factor scores than simple scale sums.

If I want to use these scales for some later analysis (like regression or ANOVA), should I just aggregate the scales so long as their reliability holds up? Or is something like CFA (testing to see if the scales hold up as good factors, which seems to be measuring the same thing as 'reliability').

I've been taught about both approaches independently and so I really don't know how they relate, whether they can be used together or which one makes more sense for which context. Is there a decision tree for good research practice in this case? Something like:

  • Run CFA according to predicted scale items

    • If CFA shows good fit, calculate factor scores and use those for analysis.
    • If CFA shows poor fit, run EFA instead and take exploratory approach (or something).

Are factor analysis and reliability testing indeed separate approaches to the same thing, or am I misunderstanding somewhere?

$\endgroup$
  • 4
    $\begingroup$ I can't tell from your 2nd paragraph, but it's worth noting that Cronbach's alpha is meaningless if there is >1 factor. $\endgroup$ – gung Jun 17 '14 at 16:19
  • 1
    $\begingroup$ Cronbach's alpha is directly related to the mean correlation among the items of the scale. It is one of the measures of item-item homogeneity. Homogeneity is one of the facets of reliability. Factor loading is the correlation between an item and the "external" criterion, the construct: even though factor was created as based on items, it is seen as external variable. A loading is thus pertains to validity, not reliability. $\endgroup$ – ttnphns Jul 22 '14 at 9:38
  • 1
    $\begingroup$ (Cont.) One should not confuse the two. Validity and reliability are partly independent, partly competitor conceptions/entities, but are not the same thing. $\endgroup$ – ttnphns Jul 22 '14 at 9:38
  • $\begingroup$ stats.stackexchange.com/q/287494/3277 is a similar question, answered. $\endgroup$ – ttnphns Jul 4 '17 at 15:14
1
$\begingroup$

I am going to add an answer here even though the question was asked a year ago. Most people who are concerned with measurement error will tell you that using factor scores from a CFA is not the best way to move forward. Doing a CFA is good. Estimating factor scores is ok as long as you correct for the amount of measurement error associated with those factor scores in subsequent analyses (a SEM program is the best place to do this).

To get the reliability of the factor score, you need to first calculate the latent construct's reliability from your CFA (or rho):

rho =  Factor score variance/(Factor score variance + Factor score standard
error^2). 

Note that the factor score standard error^2 is the error variance of the factor score. This information can be had in MPlus by requesting the PLOT3 output as part of your CFA program.

To calculate overall reliability of the factor score, you use the following formula:

(1-rho)*(FS variance+FS error variance).

The resulting value is the error variance of the factor score. If you were using MPlus for subsequent analyses, you create a latent variable defined by a single item (the factor score) and then specify the factor score's reliability:

LatentF BY FScore@1;
FScore@(calculated reliability value of factor score) 

Hope this is helpful! A great resource for this issue are the lecture notes (lecture 11, in particular) from Lesa Hoffman's SEM class at the University of Nebraska, Lincoln. http://www.lesahoffman.com/948/

$\endgroup$
  • $\begingroup$ using factor scores from a CFA is not the best way Did you mean EFA? $\endgroup$ – ttnphns Feb 27 '16 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.