2
$\begingroup$

I am comparing two regressions:

$y= \beta_0+ \beta_1 x_1 + \beta_2 x_2 + \epsilon $

and

$ y = \gamma_0 + \gamma_1 PC_1 + \gamma_2 PC_2 +\eta $.

The regressors in the second regression, $PC_1, PC_2$, are principal components generated by a PCA of $x_1,x_2$.

Presumably if the PCA just linearly transforms the original regressors, the intercepts should be identical. However, I find them very different. What is the reason behind this difference?

$\endgroup$
0
$\begingroup$

Presumably if the PCA just linearly transforms the original regressors, the intercepts should be identical.

This is not correct.

The estimate for the intercept, $\hat{\beta_0}$, can be computed as $$ \hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x_1} - \hat{\beta_2}\bar{x_2} - \dots, $$ where $\bar{y}$ denotes the sample mean of $y$, $\hat{\beta_j}$ is the sample estimate of $\beta_j$, and $\bar{x_j}$ is the sample mean of $x_j$.

So, for a simple counterexample, consider a linear transformation of $x_1$ where we just add 1 to every value. This would increase $\bar{x_1}$ by 1 and thus change the intercept.

$\endgroup$
  • $\begingroup$ Adding a constant is an affine transformation, but not a linear transformation. $\endgroup$ – David Xiaoyu Xu Aug 12 at 4:18
  • $\begingroup$ @DavidXiaoyuXu I assumed you meant linear as in f(x) = a*x + b, which adding a constant certainly is. In either case, you can see from the expression I wrote why your assumption about the intercept is incorrect $\endgroup$ – Jake Westfall Aug 12 at 4:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.