Which is more appropriate in this case -- PCA or Factor Analysis?

Question

Suppose I have an experiment that has 8 factors. These 8 factors are probably related and can hopefully be reduced. For each combination of these 8 factors that I test, I get a single output. My goal is to reduce the 8 factors to a smaller set of variables. In this case is PCA or Factor Analysis more appropriate?

I am not too clear on the differences between the two methods. My limited understanding is that PCA is best to reduce multiple outputs into a smaller set of the most important outputs and that Factor Analysis is best for finding relations between many factors. If this is true I would expect Factor Analysis to be the best method for this situation.

Answer 1

In addition to @Behacad's helpful answer, here are some of many other points that could be made.

Your data include 8 factors, which I am going to call "predictors". Otherwise we might too easily get confused with the factors yielded by factor analysis.

My shortest answer is that your goal "to reduce the 8 factors [predictors] to a smaller set of variables" is not precise enough to say which method is better for you.

Beyond that, be aware that this is a large and contentious topic. There are statistically competent authors on PCA who would not want to get enmeshed in factor analysis and regard it as oversold. The standard monographs on PCA seem close to this position. Conversely, there are statistically competent authors on factor analysis who regard PCA as just a limiting special case of factor analysis that misses the main point of factor analysis, the scope for modelling the data. And because this is a contentious topic, many people would be unhappy with my summary.

But note that PCA would yield as many components as you have predictors. Reducing them to fewer could be your choice if you decided that you could dispense with the "least important", defined somehow. Alternatively, you could use the PCA results to choose a subset of the predictors. That could be done informally, e.g. by looking at the correlations between the predictors and the PCs. Alternatively, you could check out approaches such as that explained in

Cumming, J.A. & Wooff, D.A. 2007. Dimension reduction via principal variables. Computational Statistics & Data Analysis 52(1): 550-565

It seems simplest to me to regard PCA as a multivariate transformation procedure that maps predictors to PCs, with just one main decision, to base PCs on the correlation or the covariance matrix.

Factor analysis can be carried out in exploratory or confirmatory style, the latter being very much based on modelling, which should in turn be guided by your ideas about the data generating process. It is thus best to consider factor analysis as a family of modelling methods that requires you to make choices depending on your substantive hypotheses about your data.

See also

Use of PCA analysis to select variables for a regression analysis

Answer 2

My interpretation is as follows and someone please correct me if my explanation is not useful.

Factor analysis is a good tool to reduce dimensions when you are also interested in latent variables assessed by those factors. For example, perhaps I could reduce the number of items in a measure that assesses depression (a latent construct).

Principal component analysis is useful when you want to do dimension reduction and you are less interested in latent factors. For example, perhaps you want to merely use two or three factors out of eight that best "captures" what the eight factors measure. In this case you are less interested in how these factors do or do not measure a latent construct.

There are many resources explaining the mathematical differences between these methods, but hopefully this can be useful. Tabachnick & Fidell wrote the following:

If you are interested in a theoretical solution uncontaminated by unique and error variability and have designed your study on the basis of underlying constructs that are expected to produce scores on your observed variables, FA is your choice. If, on the other hand, you simply want an empirical summary of the data set, PCA is the better choice.

Tabachnick, B. G. & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Pearson Education.