2
$\begingroup$

I am reading about ridge regression in The Elements of Statistical Learning. In the ridge regression, we do not include intercept term $\beta_0$ in the penalty term. The book says, penalization of the intercept would make the procedure depend on the origin chosen for Y; that is adding a constant $c$ to each of the targets $y_i$ would not simply result in a shift of the predictions by the same amount $c$.

Could anyone explain why this is the case? I am having hard time grasping the issue conceptually and mathematically.

Improve this question
$\endgroup$
2
  • $\begingroup$ Intuitively, I think that penalizing $\beta_0$ would never reduce the mean squared error (MSE) of a model. However, I cannot think of a nice proof. I tried looking at a simple example $y=\beta_0+\beta_1 x+\varepsilon$. The idea is to have a penalty term $\lambda_0$ on $\beta_0$ and $\lambda_1$ on $\beta_1$ and see what value of $\lambda_0$ minimizes the MSE (I expect it is $\lambda_0=0$). The tedious algebra is stopping me from actually obtaining the result... $\endgroup$
    – Richard Hardy
    Feb 14, 2015 at 16:36
  • $\begingroup$ @Richard, any non-zero lambda on any term will increase MSE, it's only the predicted MSE that is supposed to get better after regularization. $\endgroup$
    – amoeba
    Jul 16, 2015 at 7:51

0

Reset to default

Browse other questions tagged or ask your own question.