First principal component regressed against variable gives corresponding eigenvector element?

Question

I'm running into an issue where I'm noticing that the MLE for the slope ($\hat{\beta}$) in a simple linear regression between data estimated using only the first principal component and a variable used to generate that principal component is equal to the weight of that component in the linear combination. I'm wondering if this is supposed to be the case, and if so, why?

To be more clear, consider the following example (in R):

First, I generate a matrix $X$ of random normals:

test = matrix(rnorm(100), ncol=2)

Then, I perform PCA:

testpca = princomp(test)

I can see the eigenvectors as follows:

testpca[['loadings']]

Loadings:
     Comp.1 Comp.2
[1,] -0.907 -0.421
[2,]  0.421 -0.907

I only want to use the first component (let's call it $\vec{w}_1 = (-0.907, 0.421)$) and transform my original data. So, I subtract the mean and then take the dot product to get this approximation:

$(X-\bar{X}) \cdot \vec{w}_1$

In code, I perform:

centeredtest <- apply(test,2,scale,scale=FALSE,center=TRUE)
centeredtest %*% testpca[['loadings']][,1]

This is equivalent to just using the scores calculated by the PCA object:

testpca[['scores']][,1]

Now that I've applied the first principal component weights to the data, I perform a simple univariate linear regression, with the transformed data as the independent variable and the first column of the original 'test' matrix as the dependent variable. I want to scale my first 'factor' back into the original matrix space:

fit = lm(test[,1]~testpca[['scores']][,1])
summary(fit)

Coefficients:             Estimate
(Intercept)               -0.16473
testpca[["scores"]][,1]   -0.90686

And sure enough - the estimate of the slope is the first component of the eigenvector used to transform the data!

I tried to derive this, since for the univariate case $Y\sim X$ the MLE is known to be:

$\hat{\beta} = \frac{Cov(X,Y)}{Cov(X,X)}$

where, here, we have that the independent "X" is the data series: $(X-\bar{X}) \cdot \vec{w}_1$

and the dependent "Y" is just $X_{(,1)}$, but I couldn't derive that this should be $w_{1,1}$, as the computational result above seems to show!

Answer 1

I think I've got it.

Let $X$ be our data matrix, with dimensions $n\times p$.

Then, let the column vectors of $X$ be denoted as $\vec{x}_i$, where $\vec{x}_i \in\mathbb{R}^n, i=1,\ldots,p$.

After PCA, we have the principal components decomposition of $X$ as $T=XW$, where $W$ is a $p\times p$ matrix whose columns are the eigenvectors of $X^T X$.

Denote the corresponding eigenvalues $\lambda_i, i=1,\ldots,p$.

Because the loadings (columns of $T$, which we denote $\vec{t}_i$) are eigenvectors of $X^T X$, we have:

$$(X^T X) \vec{t}_1 = \lambda_1 \vec{t}_1$$

Denote the first factor to be $\vec{f}_1$; then, we have:

$$X^T \vec{f}_1 = \lambda_1 \vec{t}_1$$ $$\vec{c}_i^T \vec{f}_1 = \lambda_1 t_{(1,i)}$$

for $i=1,\ldots,p$.

Our regression model is:

$$\vec{c}_i \sim \vec{f}_1$$ $$\vec{c}_i = \beta_i \vec{f}_1 + \vec{\epsilon}_i$$

Substituting, we have:

$$\left(\beta_i\vec{f}_1 + \vec{\epsilon}_i\right)^T \vec{f}_1 = \lambda_1 t_{(1,i)}$$ $$\beta_i \vec{f}_1^T\vec{f}_1 + \vec{\epsilon}_i^T\vec{f}_1 = \lambda_1 t_{(1,i)}$$ $$\beta_i = \frac{\lambda_1 t_{(1,i)} - \left(\vec{\epsilon}_i^T \vec{f}_1\right)}{\vec{f}_1^T\vec{f}_1}$$

But, note that:

$$\vec{f}_1^T\vec{f}_1 = (A\vec{t}_1)^T(A\vec{t}_1) = \vec{t}_1^TA^TA\vec{t}_1$$

and because $\vec{t}_1$ is an eigenvector of $A^TA$ that is simply equal to:

$$\vec{f}_1^T\vec{f}_1 = \lambda_1 \vec{t}_1^T\vec{t}_1 = \lambda_1$$

so finally we arrive at:

$$\beta_i = \frac{\lambda_1 t_{(1,i)} - \left(\vec{\epsilon}_i^T \vec{f}_1\right)}{\lambda_1} = t_{(1,i)} - \left(\frac{\vec{\epsilon}_i^T\vec{f}_1}{\lambda_1}\right)$$

which, on expectation, yields the result given above.