In table 3.3 (page 63) of the elements of statistical learning book, the intercept terms for Ridge regression, lasso , pcr and PLS differ.

However, according to the theory in the book, these models should all have the same $\hat{\beta_0} = \bar{y}$. How are the intercepts estimated in the table ?

Note that: all these models are applied onto the same dataset, where the inputs are centered. enter image description here

  • 2
    $\begingroup$ Hmm.. No, they shouldn't be the same. For once, in the linear regression $y \sim N(X\beta,\sigma^2 I)$ so $\bar{y} \sim X\beta$ not $\bar{y} \sim \beta_0$. In addition to that you are talking about samples vs. population estimates. Finally specifically for this case the differences appear pretty minuet for practical purposes anyway. $\endgroup$ – usεr11852 Apr 4 '14 at 20:03
  • 1
    $\begingroup$ The intercept is an estimated mean of the $y$ for each $x$ set to zero. Each model has different $x$s. You would only get the same intercept for each model if each $x$ were centered. $\endgroup$ – AdamO Apr 4 '14 at 20:06
  • $\begingroup$ There may be a confusion here. I am not saying that intercept term for linear regression $\hat{\beta_0} =\bar{y}$. This is not what I am saying. If you try to derive $\hat{\beta_0}$ mathematically for ridge regression and lasso, you will realise that their $\hat{\beta_0} =\bar{y}$. The same goes for pcr and pls models. $\endgroup$ – mynameisJEFF Apr 5 '14 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.