prcomp() vs lm() results in R [duplicate]

Question

I have a simple matrix:

     [,1] [,2] [,3]
[1,]    1    2    3
[2,]    4    5    6
[3,]    7    8    9
[4,]   10   11   12

I have to calculate linear regression and orthogonal regression using lm() and prcomp() respectively. (for orthogonal see: here)

Assume that the first column is the the X and M the matrix I wrote before.

LINEAR REG.

mod1 <- lm(M[,1] ~ M[,2] + M[,3] + 0)

Its output is (coefficient):

Coefficients: M[, 2]  M[, 3]  
     2      -1

Ok, I have these coefficients.

Now for

ORTHOGONAL REG.

mod2 <- prcomp(~ M[,1] + M[,2] + M[,3])

Its output is:

             PC1        PC2        PC3
M[, 1] 0.5773503  0.0000000  0.8164966
M[, 2] 0.5773503 -0.7071068 -0.4082483
M[, 3] 0.5773503  0.7071068 -0.4082483

The question is: out to interpret prcomp() result instead of lm() result ? Using lm() the coefficients are using to predict the X values.

What about prcomp() ?

Thank you!

Answer 1

Although you have used $M_2$ and $M_3$ in your example, you effectively have an intercept because $M_2+1=M_3$, and so $$M_{1}=M_{2}\beta_2+M_{3}\beta_3=M_{2}\beta_2+(M_{2}+1)\beta_3=M_{2}(\beta_2+\beta_3)+\beta_3$$ (This is confirmed in @Gavin's comment, as $\beta_3=-1$ and $\beta_2+\beta_3=1$.). So your coefficient for $M_3$ is the intercept for the model with only $M_2$, and your coefficient for $M_2$ is the negative intercept for the model with only $M_3$.

In terms of a general comparison, you are comparing two straight lines (or points on straight lines). For a lm() model, we have

$$y_{i}=\beta_0+{x}_{i1}\beta_{1}+\dots+x_{ip}\beta_{p}$$

This line will pass through the point $(\beta_0,0,0,\dots,0)$ and have slopes in each direction of $(1,\beta_1,\beta_2,\dots,\beta_p)$. We can equivalently state this as passing through the point $(\beta_0+\overline{x}_1\beta_{1}+\dots+\overline{x}_p\beta_{p},\overline{x}_1,\dots,\overline{x}_p)$

Now in order to put this into the principal components analysis framework. Note that here we are only looking at the first PC. You are given a vector in $p+1$ dimensional space, $(\alpha_{Y},\alpha_{1},\dots,\alpha_{p})$, which is the principal component. This component describes a straight line, which passes through the centroid of the data $(\overline{y},\overline{x}_1,\dots,\overline{x}_p)$ and has slopes in each direction of $(\alpha_Y,\alpha_1,\dots,\alpha_p)$. Because slopes only need be defined in proportion (just alters the size of the line), we can restate this as $(1,\frac{\alpha_1}{\alpha_Y},\dots,\frac{\alpha_p}{\alpha_Y})$ (for $\alpha_Y\neq 0$ of course). This means $\beta_j$ is the equivalent quantity to $\frac{\alpha_{j}}{\alpha_Y}$. This is how you can compare the results