PCA to recover factors used during data generation. Why doesn't it work?

Question

I often found that the results of a PCA or any kind of factor analysis are interpreted in a "causal" fashion. I.e. if a principal component with high variance explanation is found, this is interpreted, that there is a common cause for all the variables that load onto this common factor. However, in a simulation, I find that other factors may be recovered, which puts some doubt on this causal interpretation. Here is the code I use for my simulation:

library(tidyverse)
N <- 10000

set.seed(298471979)

# Random variance not explained by any of the factors
v.error <- 0.01
# Remaining variance
v.g <- (1-v.error) / 2

# LATENT variables that should be recovered statistically
g1 <- rnorm(N,0,sqrt(v.g))
g2 <- rnorm(N,0,sqrt(v.g))
g3 <- rnorm(N,0,sqrt(v.g))

# Observed data caused by latent variables
y1 <- g1 + g2 + rnorm(N,0,sqrt(v.error))
y2 <- g2 + g3 + rnorm(N,0,sqrt(v.error))
y3 <- g1 + g3 + rnorm(N,0,sqrt(v.error))

Y <- cbind(y1,y2,y3)

pca <- prcomp(Y, scale. = T, center = T)

Now under the causal interpretation of the PCA results, I would expect my PCA to recover $g_1$, $g_2$, and $g_3$. However, the actual rotation matrix I get is:

Rotation (n x k) = (3 x 3):
         PC1          PC2        PC3
y1 0.5785666 -0.002218839  0.8156321
y2 0.5767353  0.708224186 -0.4071792
y3 0.5767469 -0.705984121 -0.4110346

Which indicates a common underlying cause for $y_1$, $y_2$, and $y_3$ (which is not what I actually generated). My first guess, was that the $g_i$ may not be perfectly uncorrelated (in fact they are not) and hence the PCA is trying to account for this slight correlation. However, a theoretical analysis showed that I would get a similar rotation matrix even for perfectly uncorrelated $g_i$.

In fact, if I do the same thing theoretical and define

$y_1=g_1+g_2+\epsilon_1$ with $\epsilon_1 \sim N(0,\sigma^2)$

and similar for $y_2$ and $y_3$, I find that if I try to recover $g_1$ as

$g'_1=(y_1-y_2+y_3)/2$

and $g_2$ as

$g'_2=(y_1+y_2-y_3)/2$

then I have a covariance between $g'_1$ and $g'_2$ that can be calculated to be

$Cov[g'_1,g'_2]=\sigma^2/4.$

So the recovered variables are in fact not orthogonal. However, if I manually do the eigenvalue computation, I get the eigenvector structure of

$\left\{\pmatrix{1\\1\\1\\},\pmatrix{0\\1\\-1\\},\pmatrix{1\\-1/2\\-1/2\\}\right\}$

and this closely matches the rotation matrix of the PCA.

So when does this causal interpretation of a PCA actually make sense? Or is this interpretation always incorrect, similar to mixing up correlation and causation? Also on a related note, are there exact conditions when a PCA might be able to find the original factors?

EDIT:

Sorry for not clearly distinguishing between "factors" and "components". Since both are often treated very similarly, I also tend to mix them up from time to time. My question, however, can partially be related to both.

The situation I am trying to describe is where there are three (unobservable) latent variable $g_1$, $g_2$, $g_3$. However, these are not observable independenlty, but only as their combinations $y_1$, $y_2$, and $y_3$. Since the $y_i$ result from the $g_i$ it should be possible to reconstruct those.

I am aware, that it is possible to use a PCA to decorrelate data (by multiplying with the rotation matrix) and then reconstruct the original (observable) data back from the decorrelated variables (by multiplying with the transposed rotation matrix). But this is not what I am after, I would like to retrieve the latent unobservable variables.

A true factor analysis (using factanal), in this case, produces very similar results:

Loadings:
   Factor1
y1 0.710  
y2 0.703  
y3 0.703 

which almost matches the first component I found using a PCA. Also factanal wont let me retrieve more than one factor from $Y$.

Answer 1

PCA is not a method of causal inference in the sense of recovering a data generating model. All PCA does is decompose a covariance matrix into orthogonal vectors that define a new space with the property that if the data are "projected" onto this space, then each successive axis has less variance. An infinite number of generating models can generate the same covariance matrix and consequently the eigenvectors of a covariance matrix cannot recover the true generating model. Your generating model has three variables (g1, g2, and g3) that all contribute equally to the variation in the data and your PCA recovered this quite well (as the first axis), as it should.

Answer 2

I'm not sure if I understand your entire question, but my answer's too long for a comment, so i put it here anyway. Hope it helps. Sorry if I miss the point.

So when does this causal interpretation of a PCA actually make sense?

You use the term PCA and then the term 'factors', which is often used to refer to factor analysis. The two are similar but come from totally different points of views. They are not interchangeable.

PCA, in stats anyway, is not regarded as causal at all (unless there's some really obscure way of using it as such). It is generally viewed as a dimension reduction procedure. The principal components are orthogonal and each successive one will account for less and less variance, so you just drop the last few and say that you've reduced the dimensions but kept most of the structure of the data in tact.

Factor analysis can be viewed as causal, sort of. Or at least, that the correlation of observed variables are because of underlying unobserved latent variables--the factors. Factor analysis explicitly assumes that the loadings are correlations between the standardized observed variables and the underlying unobserved factor. PCA doesn't need this at all. Often you will assume factors are uncorrelated for simplicity but they don't have to be, whereas principal components are by definition orthogonal.

After factor analysis, you will use Structural Equation Modelling to establish causation between latent factors and each other.

Random variance not explained by any of the factors

You had this comment in your code, PCA does have any unique variance that will not be accounted for by the principal components. If you have as many principal components as you have variables they will account for all variance in the original data. Factor analysis on the other hand, does have unique variance of each variable that will not load entirely onto the factors.

Which indicates a common underlying cause for y1, y2, and y3 (which is not what I actually generated).

It looks like you did input Y. From your code, it looks like you generated Y from G, and cbined them into a matrix. You then fed that matrix into the prcomp command. So, R is using the covariance matrix of Y's, not of G's. So the principal components are in terms of Y's.

If you cbind(g1, g2, g3) and the feed that into prcomp you should get principal components in terms of G.

My first guess, was that the gi may not be perfectly uncorrelated (in fact they are not) and hence the PCA is trying to account for this slight correlation.

You definitely want your original variables to be correlated, not uncorrelated. The point is to rotate the matrix so that it maximizes the variance of the data onto the first PC, and then maximizes all the variance remaining onto the next PC and so on. If the data is not correlated then the correlation matrix is literally the identity matrix and all the eigenvalues would equal 1, and so it won't do anything. Here's another answer of mine on this exact topic.

Also on a related note, are there exact conditions when a PCA might be able to find the original factors?

I'm not sure if the following is what you mean, but if you don't drop any principal components then you can just unrotate the data and you have your original values again. In your case, you can rotate the data back to get Y and then solve for G again. Here's an awesome answer of how to do that using different codes.