I often found that the results of a PCA or any kind of factor analysis are interpreted in a "causal" fashion. I.e. if a principal component with high variance explanation is found, this is interpreted, that there is a common cause for all the variables that load onto this common factor. However, in a simulation, I find that other factors may be recovered, which puts some doubt on this causal interpretation. Here is the code I use for my simulation:
library(tidyverse) N <- 10000 set.seed(298471979) # Random variance not explained by any of the factors v.error <- 0.01 # Remaining variance v.g <- (1-v.error) / 2 # LATENT variables that should be recovered statistically g1 <- rnorm(N,0,sqrt(v.g)) g2 <- rnorm(N,0,sqrt(v.g)) g3 <- rnorm(N,0,sqrt(v.g)) # Observed data caused by latent variables y1 <- g1 + g2 + rnorm(N,0,sqrt(v.error)) y2 <- g2 + g3 + rnorm(N,0,sqrt(v.error)) y3 <- g1 + g3 + rnorm(N,0,sqrt(v.error)) Y <- cbind(y1,y2,y3) pca <- prcomp(Y, scale. = T, center = T)
Now under the causal interpretation of the PCA results, I would expect my PCA to recover $g_1$, $g_2$, and $g_3$. However, the actual rotation matrix I get is:
Rotation (n x k) = (3 x 3): PC1 PC2 PC3 y1 0.5785666 -0.002218839 0.8156321 y2 0.5767353 0.708224186 -0.4071792 y3 0.5767469 -0.705984121 -0.4110346
Which indicates a common underlying cause for $y_1$, $y_2$, and $y_3$ (which is not what I actually generated). My first guess, was that the $g_i$ may not be perfectly uncorrelated (in fact they are not) and hence the PCA is trying to account for this slight correlation. However, a theoretical analysis showed that I would get a similar rotation matrix even for perfectly uncorrelated $g_i$.
In fact, if I do the same thing theoretical and define
$y_1=g_1+g_2+\epsilon_1$ with $\epsilon_1 \sim N(0,\sigma^2)$
and similar for $y_2$ and $y_3$, I find that if I try to recover $g_1$ as
and $g_2$ as
then I have a covariance between $g'_1$ and $g'_2$ that can be calculated to be
So the recovered variables are in fact not orthogonal. However, if I manually do the eigenvalue computation, I get the eigenvector structure of
and this closely matches the rotation matrix of the PCA.
So when does this causal interpretation of a PCA actually make sense? Or is this interpretation always incorrect, similar to mixing up correlation and causation? Also on a related note, are there exact conditions when a PCA might be able to find the original factors?
Sorry for not clearly distinguishing between "factors" and "components". Since both are often treated very similarly, I also tend to mix them up from time to time. My question, however, can partially be related to both.
The situation I am trying to describe is where there are three (unobservable) latent variable $g_1$, $g_2$, $g_3$. However, these are not observable independenlty, but only as their combinations $y_1$, $y_2$, and $y_3$. Since the $y_i$ result from the $g_i$ it should be possible to reconstruct those.
I am aware, that it is possible to use a PCA to decorrelate data (by multiplying with the rotation matrix) and then reconstruct the original (observable) data back from the decorrelated variables (by multiplying with the transposed rotation matrix). But this is not what I am after, I would like to retrieve the latent unobservable variables.
A true factor analysis (using
factanal), in this case, produces very similar results:
Loadings: Factor1 y1 0.710 y2 0.703 y3 0.703
which almost matches the first component I found using a PCA. Also
factanal wont let me retrieve more than one factor from $Y$.