What are the differences between Factor Analysis and Principal Component Analysis

Question

It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use one over the other. A real example would be incredibly useful.

Answer 1

Principal components analysis involves extracting linear composites of observed variables.

Factor analysis is based on a formal model predicting observed variables from theoretical latent factors.

In psychology these two techniques are often applied in the construction of multi-scale tests to determine which items load on which scales. They typically yield similar substantive conclusions (for a discussion see Comrey (1988) Factor-Analytic Methods of Scale Development in Personality and Clinical Psychology). This helps to explain why some statistics packages seem to bundle them together. I have also seen situations where "principal component analysis" is incorrectly labelled "factor analysis".

In terms of a simple rule of thumb, I'd suggest that you:

1. Run factor analysis if you assume or wish to test a theoretical model of latent factors causing observed variables.

2. Run principal components analysis If you want to simply reduce your correlated observed variables to a smaller set of important independent composite variables.

Answer 2

From my response here:

http://stats.stackexchange.com/questions/612/is-psychprincipal-function-still-pca-when-using-rotation

Principal Components Analysis (PCA) and Common Factor Analysis (CFA) are distinct methods. Often, they produce similar results and PCA is used as the default extraction method in the SPSS Factor Analysis routines. This undoubtedly results in a lot of confusion about the distinction between the two.

The bottom line is, these are two different models, conceptually. In PCA, the components are actual orthogonal linear combinations that maximize the total variance. In FA, the factors are linear combinations that maximize the shared portion of the variance--underlying "latent constructs". That's why FA is often called "common factor analysis". FA uses a variety of optimization routines and the result, unlike PCA, depends on the optimization routine used and starting points for those routines. Simply there is not a single unique solution.

In R, the factanal() function provides CFA with a maximum likelihood extraction. So, you shouldn't expect it to reproduce an SPSS result which is based on a PCA extraction. It's simply not the same model or logic. I'm not sure if you would get the same result if you used SPSS's Maximum Likelihood extraction either as they may not use the same algorithm.

For better or for worse in R, you can, however, reproduce the mixed up "factor analysis" that SPSS provides as its default. Here's the process in R. With this code, I'm able to reproduce the SPSS Principal Component "Factor Analysis" result using this dataset. (With the exception of the sign, which is indeterminant). That result could also then be rotated using any of Rs available rotation methods.

# Load the base dataset attitude to work with.
data(attitude)
# Compute eigenvalues and eigen vectors of the correlation matrix.
pfa.eigen<-eigen(cor(attitude))
# Print and note that eigen values are those produced by SPSS.
# Also note that SPSS will extract 2 components as eigen values > 1 = 2
pfa.eigen$values
# set a value for the number of factors (for clarity)
factors<-2
# Extract and transform two components.
pfa.eigen$vectors [ , 1:factors ]  %*% 
+ diag ( sqrt (pfa.eigen$values [ 1:factors ] ),factors,factors )

Answer 3

You are right about your first point, although in FA you generally work with both (uniqueness and communality). The choice between PCA and FA is a long-standing debate among psychometricians. I don't quite follow your points, though. Rotation of principal axes can be applied whatever the method is used to constructed latent factors. In fact, most of the times this is the VARIMAX rotation (orthogonal rotation, considering uncorrelated factors) that is used, for practical reasons (easiest interpretation, easiest scoring rules or interpretation of factor scores, etc.), although oblique rotation (e.g. PROMAX) might probably better reflect the reality (latent constructs are often correlated one each other), at least in the tradition of FA where you assume that a latent construct is really at the heart of the observed inter-correlations between your variables. The point is that PCA followed by VARIMAX rotation somewhat distorts the interpretation of the linear combinations of the original variables in the "data analysis" tradition (see the work of Michel Tenenhaus). From a psychometrical perspectice, FA models are to be preferred since they explicitely account for measurement errors, while PCA doesn't care about that. Briefly stated, using PCA you are expressing each component (factor) as a linear combination of the variables, whereas in FA these are the variables that are expressed as linear combinations of the factors (including communalities and uniqueness components, as you said).

I recommend you to read first the following discussions about this topic:

• What are the differences between Factor Analysis and Principal Component Analysis
• On the use of oblique rotation after PCA -- see reference therein

Answer 4

There are numerous suggested definitions on the web. Here is one from a on-line glossary on statistical learning:

Principal Component Analysis

Constructing new features which are the principal components of a data set. The principal components are random variables of maximal variance constructed from linear combinations of the input features. Equivalently, they are the projections onto the principal component axes, which are lines that minimize the average squared distance to each point in the data set. To ensure uniqueness, all of the principal component axes must be orthogonal. PCA is a maximum-likelihood technique for linear regression in the presence of Gaussian noise on both inputs and outputs. In some cases, PCA corresponds to a Fourier transform, such as the DCT used in JPEG image compression. See "Eigenfaces for recognition" (Turk&Pentland, J Cognitive Neuroscience 3(1), 1991), Bishop, "Probabilistic Principal Component Analysis", and "Automatic choice of dimensionality for PCA".choice of dimensionality for PCA".

Factor analysis

A generalization of PCA which is based explicitly on maximum-likelihood. Like PCA, each data point is assumed to arise from sampling a point in a subspace and then perturbing it with full-dimensional Gaussian noise. The difference is that factor analysis allows the noise to have an arbitrary diagonal covariance matrix, while PCA assumes the noise is spherical. In addition to estimating the subspace, factor analysis estimates the noise covariance matrix. See "The EM Algorithm for Mixtures of Factor Analyzers".choice of dimensionality for PCA".

Answer 5

Differences between factor analysis and principal components analysis are:

• In factor analysis there is a structured model and some assumptions. In this respect it is a statistical technique which does not apply to principal components analysis which is a purely mathematical transformation.

• The aim of principal components analysis is to explain the variance while factor analysis explains the covariance between the variables.

One of the biggest reasons for the confusion between the two, has to do with the fact that one of the factor extraction methods in Factor Analysis is called "method of principal components". However, it's one thing to use PCA and another thing to use the method of principal components in FA. The names maybe similar, but there are significant differences. The former is an independent analytical method while the latter is merely a tool for factor extraction.

Answer 6

For me (and i hope this is useful) factor analysis is much more useful than PCA.

Recently, i had the pleasure of analysing a scale through factor analysis. This scale (although its widely used in industry) was developed by using PCA, and to my knowledge had never been factor analysed.

When i performed the factor analysis (principal axis) i discovered that the communalities for three of the items were less than 30%, which means that over 70% of the items variance was not being analysed. PCA just transforms the data into a new combination and doesnt care about communalities. My conclusion was that the scale was not a very good one from a psychometric point of view, and i've confirmed this with a different sample.

Essentially, if you want to predict using the factors, use PCA, while if you want to understand the latent factors, use Factor Analysis.

Answer 7

On can think of a PCA as being like a FA in which the communalities are assumed to equal 1 for all variables. In practice, this means that items that would have relatively low factor loadings in FA due to low communality will have higher loadings in PCA. This is not a desirable feature if the primary purpose of the analysis is to cut item length and clean a battery of items of those with low or equivocal loadings, or to identify concepts that are not well represented in the item pool.

Best,

Stephen Brand

www.StatisticsDoc.com