I'm running into an issue where I'm noticing that the MLE for the slope ($\hat{\beta}$) in a simple linear regression between data estimated using only the first principal component and a variable used to generate that principal component is equal to the weight of that component in the linear combination. I'm wondering if this is supposed to be the case, and if so, why?
To be more clear, consider the following example (in R):
First, I generate a matrix $X$ of random normals:
test = matrix(rnorm(100), ncol=2)
Then, I perform PCA:
testpca = princomp(test)
I can see the eigenvectors as follows:
testpca[['loadings']]
Loadings:
Comp.1 Comp.2
[1,] -0.907 -0.421
[2,] 0.421 -0.907
I only want to use the first component (let's call it $\vec{w}_1 = (-0.907, 0.421)$) and transform my original data. So, I subtract the mean and then take the dot product to get this approximation:
$(X-\bar{X}) \cdot \vec{w}_1$
In code, I perform:
centeredtest <- apply(test,2,scale,scale=FALSE,center=TRUE)
centeredtest %*% testpca[['loadings']][,1]
This is equivalent to just using the scores calculated by the PCA object:
testpca[['scores']][,1]
Now that I've applied the first principal component weights to the data, I perform a simple univariate linear regression, with the transformed data as the independent variable and the first column of the original 'test' matrix as the dependent variable. I want to scale my first 'factor' back into the original matrix space:
fit = lm(test[,1]~testpca[['scores']][,1])
summary(fit)
Coefficients: Estimate
(Intercept) -0.16473
testpca[["scores"]][,1] -0.90686
And sure enough - the estimate of the slope is the first component of the eigenvector used to transform the data!
I tried to derive this, since for the univariate case $Y\sim X$ the MLE is known to be:
$\hat{\beta} = \frac{Cov(X,Y)}{Cov(X,X)}$
where, here, we have that the independent "X" is the data series: $(X-\bar{X}) \cdot \vec{w}_1$
and the dependent "Y" is just $X_{(,1)}$, but I couldn't derive that this should be $w_{1,1}$, as the computational result above seems to show!