19
$\begingroup$

I would like to know if it makes any logical sense to perform principal component analysis (PCA) and exploratory factor analysis (EFA) on the same data set. I have heard professionals expressly recommend:

  1. Understand what the goal of analysis is and choose PCA or EFA for the data analysis;
  2. Having done one analysis there is no need to do the other analysis.

I understand the motivational differences between the two, but I was just wondering if there is anything wrong in interpreting the results provided by PCA and EFA at the same time?

$\endgroup$
  • 8
    $\begingroup$ Why the insecurity? If you understand the motivational differences between the two, you should be in one of two positions: Regard them as complementary and be willing to explore both. Regard one as much more convincing for what you want to do. It seems that you want to be told that there's a right thing to do, but PCA with or versus FA is such a longstanding area of controversy that if two experts agree it is usually only that they both disagree with a third expert, but for different reasons. $\endgroup$ – Nick Cox Apr 16 '14 at 19:52
  • $\begingroup$ What are you studying? Some social sciences thing like happiness or objective data like interest rates? $\endgroup$ – Aksakal Jan 16 '15 at 16:25
20
$\begingroup$

Both models - principal-component and common-factor - are similar straightforward linear regressional models predicting observed variables by latent variables. Let us have centered variables V1 V2 ... Vp and we chose to extract 2 components/factors FI and FII. Then the model is the system of equations:

$V_1 = a_{1I}F_I + a_{1II}F_{II} + E_1$

$V_2 = a_{2I}F_I + a_{2II}F_{II} + E_2$

$...$

$V_p = …$

where coefficient a is a loading, F is a factor or a component, and variable E is regression residuals. Here, FA model differs from PCA model exactly by that FA imposes the requirement: variables E1 E2 ... Ep (the error terms which are uncorrelated with the Fs) must not correlate with each other (See pictures) . These error variables FA calls "unique factors"; their variances are known ("uniquenesses") but their casewise values are not. Therefore, factor scores F are computed as good approximations only, they are not exact.

(A matrix algebra presentation of this common factor analysis model is in Footnote $^1$.)

Whereas in PCA the error variables from predicting different variables may freely correlate: nothing is imposed on them. They represent that "dross" we've taken the left-out p-2 dimensions for. We know the values of E and so we can compute component scores F as exact values.

That was the difference between PCA model and FA model.

It is due to that above outlined difference, that FA is able to explain pairwise correlations (covariances). PCA generally cannot do it (unless the number of extracted components = p); it can only explain multivariate variance$^2$. So, as long as "Factor analysis" term is defined via the aim to explain correlations, PCA is not factor analysis. If "Factor analysis" is defined more broadly as a method providing or suggesting latent "traits" which could be interpreted, PCA can be seen is a special and simplest form of factor analysis.

Sometimes - in some datasets under certain conditions - PCA leaves E terms which almost do not intercorrelate. Then PCA can explain correlations and become like FA. It is not very uncommon with datasets with many variables. This made some observers to claim that PCA results become close to FA results as data grows. I don't think it is a rule, but the tendency may indeed be. Anyway, given their theoretical differences, it is always good to select the method consciously. FA is a more realistic model if you want to reduce variables down to latents which you're going to regard as real latent traits standing behind the variables and making them correlate.

But if you have another aim - reduce dimensionality while keeping the distances between the points of the data cloud as much as possible - PCA is better than FA. (However, iterative Multidimensional scaling (MDS) procedure will be even better then. PCA amounts to noniterative metric MDS.) If you further don't bother with the distances much and are interested only in preserving as much of the overall variance of the data as possible, by few dimensions - PCA is an optimal choice.


$^1$ Factor analysis data model: $\mathbf {V=FA'+E}diag \bf(u)$, where $\bf V$ is n cases x p variables analyzed data (columns centered or standardized), $\bf F$ is n x m common factor values (the unknown true ones, not factor scores) with unit variance, $\bf A$ is p x m matrix of common factor loadings (pattern matrix), $\bf E$ is n x p unique factor values (unknown), $\bf u$ is the p vector of the unique factor loadings equal to the sq. root of the uniquenesses ($\bf u^2$). Portion $\mathbf E diag \bf(u)$ could be just labeled as "E" for simplicity, as it is in the formulas opening the answer.

Principal assumptions of the model:

  • $\bf F$ and $\bf E$ variables (common and unique factors, respectively) have zero means and unit variances; $\bf E$ is typically assumed multivariate normal but $\bf F$ in general case needs not be multivariate normal (if both are assumed multivariate normal then $\bf V$ are so, too);
  • $\bf E$ variables are uncorrelated with each other and are uncorrelated with $\bf F$ variables.

$^2$ It follows from the common factor analysis model that loadings $\bf A$ of m common factors (m<p variables), also denoted $\bf A_{(m)}$, should closely reproduce observed covariances (or correlations) between the variables, $\bf \Sigma$. So that if factors are orthogonal, the fundamental factor theorem states that

$\bf \hat{\Sigma} = AA'$ and $\bf \Sigma \approx \hat{\Sigma} + \it diag \bf (u^2)$,

where $\bf \hat{\Sigma}$ is the matrix of reproduced covariances (or correlations) with common variances ("communalities") on its diagonal; and unique variances ("uniquenesses") - which are variances minus communalities - are the vector $\bf u^2$. The off-diagonal discrepancy ($\approx$) is due to that factors is a theoretical model generating data, and as such it is simpler than the observed data it was built on. The main causes of the discrepancy between the observed and the reproduced covariances (or correlations) may be: (1) number of factors m is not statistically optimal; (2) partial correlations (these are p(p-1)/2 factors that do not belong to common factors) are pronounced; (3) communalities not well assesed, their initial values had been poor; (4) relationships are not linear, using linear model is questionnable; (5) model "subtype" produced by the extraction method is not optimal for the data (see about different extraction methods). In other words, some FA data assumptions are not fully met.

As for plain PCA, it reproduces covariances by the loadings exactly when m=p (all components are used) and it usually fails to do it if m<p (only few 1st components retained). Factor theorem for PCA is:

$\bf \Sigma= AA'_{(p)} = AA'_{(m)} + AA'_{(p-m)}$,

so both $\bf A_{(m)}$ loadings and dropped $\bf A_{(p-m)}$ loadings are mixtures of communalities and uniquenesses and neither individually can help restore covariances. The closer m is to p, the better PCA restores covariances, as a rule, but small m (which often is of our interest) don't help. This is different from FA, which is intended to restore covariances with quite small optimal number of factors. If $\bf AA'_{(p-m)}$ approaches diagonality PCA becomes like FA, with $\bf A_{(m)}$ restoring all the covariances. It happens occasionally with PCA, as I've already mentioned. But PCA lacks algorithmic ability to force such diagonalization. It is FA algorithms who do it.

FA, not PCA, is a data generative model: it presumes few "true" common factors (of usually unknown number, so you try out m within a range) which generate "true" values for covariances. Observed covariances are the "true" ones + small random noise. (It is due to the performed diagonalization that leaved $\bf A_{(m)}$ the sole restorer of all covariances, that the above noise can be small and random.) Trying to fit more factors than optimal amounts to overfitting attempt, and not necessarily efficient overfitting attempt.

Both FA and PCA aim to maximize $trace(\bf A'A_{(m)})$, but for PCA it is the only goal; for FA it is the concomitant goal, the other being to diagonalize off uniquenesses. That trace is the sum of eigenvalues in PCA. Some methods of extraction in FA add more concomitant goals at the expense of maximizing the trace, so it is not of principal importance.

To summarize the explicated differences between the two methods. FA aims (directly or indirectly) at minimizing differences between individual corresponding off-diagonal elements of $\bf \Sigma$ and $\bf AA'$. A successful FA model is the one that leaves errors for the covariances small and random-like (normal or uniform about 0, no outliers/fat tails). PCA only maximizes $trace(\bf AA')$ which is equal to $trace(\bf A'A)$ (and $\bf A'A$ is equal to the covariance matrix of the principal components, which is diagonal matrix). Thus PCA isn't "busy" with all the individual covariances: it simply cannot, being merely a form of orthogonal rotation of data.

Thanks to maximizing the trace - the variance explained by m components - PCA is accounting for covariances, since covariance is shared variance. In this sense PCA is "low-rank approximation" of the whole covariance matrix of variables. And when seen from the viewpoint of observations this approximation is the approximation of the Euclidean-distance matrix of observations (which is why PCA is metric MDS called "Principal coordinate analysis). This fact should not screen us from the reality that PCA does not model covariance matrix (each covariance) as generated by few living latent traits that are imaginable as transcendent towards our variables; the PCA approximation remains immanent, even if it is good: it is simplification of the data.


If you want to see step-by-step computations done in PCA and FA, commented and compared, please look in here.

$\endgroup$
  • $\begingroup$ It is an excellent answer. $\endgroup$ – Subhash C. Davar Apr 24 '14 at 16:46
  • 2
    $\begingroup$ +1 for bringing me a fresh perspective of viewing PCA. Now as I understand it, both PCA and FA can explain variance of the observed variables, and since FA dictates that the error terms for each variable should not be correlated, whereas PCA does not make such dictation, so FA can capture all the covariance in the observed variables, but PCA fails to do that, because in PCA the error terms might also contain some covariance of the observed variables, unless we use all the PC to represent the observed variables, right? $\endgroup$ – avocado Apr 25 '14 at 3:36
  • 1
    $\begingroup$ Exactly. PCA can not only underestimate a covariance value (as you may probably think), it can also overestimate it. In short, a1*a2<>Cov12, which is normal behaviour for PCA. For FA, that would be the sign of suboptimal solution (e.g., wrong number of factors extracted). $\endgroup$ – ttnphns Apr 25 '14 at 6:41
  • $\begingroup$ @ttnphns: +1, but I am quite confused. I am fairly well familiar with PCA, but know very little about FA. My understanding was that in PCA the covariance matrix is decomposed as $\Sigma = WW^\top+\sigma^2 I$ and in FA as $\Sigma = WW^\top+\Psi$ with diagonal $\Psi$, i.e. PCA assumes isotropic noise covariance and FA --- diagonal one. That's how it is stated in Bishop's textbook and in all treatments on probabilistic PCA (PPCA) that I came across. Crucially, in both cases the noise covariance is diagonal, i.e. noise terms do not correlate. How can I reconcile it with what you wrote here? $\endgroup$ – amoeba says Reinstate Monica Apr 26 '14 at 21:58
  • $\begingroup$ @amoeba, off-diagonal elements of $\Sigma$ are theoretically restored in FA by $WW'$ ($W$=the loadings), but they are not restored theoretically by it in plain PCA (and so $\Psi$ isn't diagonal, but is square, in PCA). I didn't study PPCA, sorry (is it possible to download Bishop's book?). What is $\sigma^2$ in your notation? And please define "isotropic". $\endgroup$ – ttnphns Apr 27 '14 at 2:43
6
$\begingroup$

I provided my own account of the similarities and differences between PCA and FA in the following thread: Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis?

Note that my account is somewhat different from the one by @ttnphns (as presented in his answer above). My main claim is that PCA and FA are not as different as is often thought. They can indeed strongly differ when the number of variables is very low, but tend to yield quite similar results once the number of variables is over around a dozen. See my [long!] answer in the linked thread for mathematical details and Monte Carlo simulations. For a much more concise version of my argument see here: Under which conditions do PCA and FA yield similar results?

Here I would like to explicitly answer your main question: Is there anything wrong with performing PCA and FA on the same data set? My answer to this is: No.

When running PCA or FA, you are not testing any hypothesis. Both of them are exploratory techniques that are used to get a better understanding of the data. So why not explore the data with two different tools? In fact, let's do it!

Example: wine data set

As an illustration, I used a fairly well-known wine dataset with $n=178$ wines from three different grapes described by $p=13$ variables. See my answer here: What are the differences between Factor Analysis and Principal Component Analysis? for mode details, but briefly -- I ran both PCA and FA analysis and made 2D biplots for both of them. One can easily see that the difference is minimal:

PCA and FA analysis of the wine dataset

$\endgroup$
  • $\begingroup$ If the results turn out to be very similar, then you can decide to stick with only one approach. Sure. How much similar then? If the results turn out to be very different, then maybe it tells you something about your data That's perfectly mystic & esoteric. $\endgroup$ – ttnphns Jan 16 '15 at 18:35
  • $\begingroup$ Hmmm, sorry if it was unclear. What I meant is that if there are many variables and PCA yields very different loadings from FA, it tells us something. Perhaps, communalities are very low (i.e. correlation matrix is dominated by the diagonal, and off-diagonal elements are small). This can be an interesting observation. If I for some reason analyzed the same dataset with PCA and FA and got very different results, I would investigate it further. Does it make sense? $\endgroup$ – amoeba says Reinstate Monica Jan 16 '15 at 19:51
  • $\begingroup$ @ttnphns: I made an update with a worked-out example for one particular dataset. I hope you will enjoy it! See also my linked (new) answer. It's the first time I made a FA biplot, and our earlier conversations helped me a lot for that. $\endgroup$ – amoeba says Reinstate Monica Jan 17 '15 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.