2
$\begingroup$

Suppose I have an experiment that has 8 factors. These 8 factors are probably related and can hopefully be reduced. For each combination of these 8 factors that I test, I get a single output. My goal is to reduce the 8 factors to a smaller set of variables. In this case is PCA or Factor Analysis more appropriate?

I am not too clear on the differences between the two methods. My limited understanding is that PCA is best to reduce multiple outputs into a smaller set of the most important outputs and that Factor Analysis is best for finding relations between many factors. If this is true I would expect Factor Analysis to be the best method for this situation.

$\endgroup$
2
  • 2
    $\begingroup$ Did you read through the following thread: What are the differences between FA & PCA? $\endgroup$ Jun 24 '13 at 22:41
  • 2
    $\begingroup$ @OSE - in case you are disinclined to follow gung's pointer, - FA explains, by a few factors, pairwise covariances (or correlations) between the variables. PCA explains, by a few components, the bulk of the multivariate variance. PCA may explain covariances too, sometimes, but it's not guaranteed. $\endgroup$
    – ttnphns
    Jun 25 '13 at 0:22
3
$\begingroup$

In addition to @Behacad's helpful answer, here are some of many other points that could be made.

Your data include 8 factors, which I am going to call "predictors". Otherwise we might too easily get confused with the factors yielded by factor analysis.

My shortest answer is that your goal "to reduce the 8 factors [predictors] to a smaller set of variables" is not precise enough to say which method is better for you.

Beyond that, be aware that this is a large and contentious topic. There are statistically competent authors on PCA who would not want to get enmeshed in factor analysis and regard it as oversold. The standard monographs on PCA seem close to this position. Conversely, there are statistically competent authors on factor analysis who regard PCA as just a limiting special case of factor analysis that misses the main point of factor analysis, the scope for modelling the data. And because this is a contentious topic, many people would be unhappy with my summary.

But note that PCA would yield as many components as you have predictors. Reducing them to fewer could be your choice if you decided that you could dispense with the "least important", defined somehow. Alternatively, you could use the PCA results to choose a subset of the predictors. That could be done informally, e.g. by looking at the correlations between the predictors and the PCs. Alternatively, you could check out approaches such as that explained in

Cumming, J.A. & Wooff, D.A. 2007. Dimension reduction via principal variables. Computational Statistics & Data Analysis 52(1): 550-565

It seems simplest to me to regard PCA as a multivariate transformation procedure that maps predictors to PCs, with just one main decision, to base PCs on the correlation or the covariance matrix.

Factor analysis can be carried out in exploratory or confirmatory style, the latter being very much based on modelling, which should in turn be guided by your ideas about the data generating process. It is thus best to consider factor analysis as a family of modelling methods that requires you to make choices depending on your substantive hypotheses about your data.

See also

Use of PCA analysis to select variables for a regression analysis

$\endgroup$
1
  • $\begingroup$ Thanks for adding to this. I did not want to oversimplify things! $\endgroup$
    – Behacad
    Jun 24 '13 at 23:42
2
$\begingroup$

My interpretation is as follows and someone please correct me if my explanation is not useful.

Factor analysis is a good tool to reduce dimensions when you are also interested in latent variables assessed by those factors. For example, perhaps I could reduce the number of items in a measure that assesses depression (a latent construct).

Principal component analysis is useful when you want to do dimension reduction and you are less interested in latent factors. For example, perhaps you want to merely use two or three factors out of eight that best "captures" what the eight factors measure. In this case you are less interested in how these factors do or do not measure a latent construct.

There are many resources explaining the mathematical differences between these methods, but hopefully this can be useful. Tabachnick & Fidell wrote the following:

If you are interested in a theoretical solution uncontaminated by unique and error variability and have designed your study on the basis of underlying constructs that are expected to produce scores on your observed variables, FA is your choice. If, on the other hand, you simply want an empirical summary of the data set, PCA is the better choice.

Tabachnick, B. G. & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Pearson Education.

$\endgroup$
2
  • $\begingroup$ I will also add that likely neither option is wrong and both are quite defensible. Perhaps pick the one you are most familiar with. I would probably do PCA in this case, although I prefer FA in most cases. $\endgroup$
    – Behacad
    Jun 24 '13 at 23:44
  • $\begingroup$ One minor flaw of your otherwise good answer, @Behacad, is that p. components can be called "latent constructs" also. So using this term to distinguish FA and PCA without explaining substantial model differences helps little. $\endgroup$
    – ttnphns
    Jun 25 '13 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.