0
$\begingroup$

I have come across two different ways of obtaining the ridge regression solution, which are as follows:

Method1:-(obtained from here)
$RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\Omega\beta$
In this case,

  1. $X$ and $Y$ need not be centered
  2. $X$ contains the column of ones. i.e. $X$ has dimensions $N\times (p+1)$, where $N$ is the number of data points and $p$ is the dimension of each data point.
  3. $$\beta=[ \space\space\beta_0 \space\space\space \beta_1 \space\space\space \beta_2 \space\space\space \cdots\space\space\space \beta_p\space\space]^T \text{ and } \Omega=\text{diag}(0,1,1,\cdots ,1)\implies \beta^T\Omega\beta=\sum_{i=1}^p\beta_i^2$$

$\implies RSS(\beta)=Y^TY-2Y^TX\beta+\beta^T(X^TX+\lambda\Omega)\beta$

$\implies \dfrac{\partial}{\partial\beta}RSS(\beta)=0\implies\hat{\beta}_{ridge}=(X^TX+\lambda\Omega)^{-1}X^TY=[ \space\space\hat{\beta_0} \space\space\space \hat{\beta_1} \space\space\space \hat{\beta_2} \space\space\space \cdots\space\space\space \hat{\beta_p} \space\space]^T$

Method2:-
$RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\beta$
In this case,

  1. $X$ and $Y$ need to be centered
  2. $X$ does not contains the column of ones. i.e. $X$ has dimensions $N\times p$, where $N$ is the number of data points and $p$ is the dimension of each data point.
  3. $$\beta=[ \space\space \beta_1 \space\space\space \beta_2 \space\space\space \cdots\space\space\space \beta_p\space\space]^T \implies \beta^T\beta=\sum_{i=1}^p\beta_i^2$$

$\implies RSS(\beta)=Y^TY-2Y^TX\beta+\beta^T(X^TX+\lambda I)\beta$

$\implies \dfrac{\partial}{\partial\beta}RSS(\beta)=0\implies\hat{\beta}_{ridge}=(X^TX+\lambda I)^{-1}X^TY=[ \space\space \hat{\beta_1} \space\space\space \hat{\beta_2} \space\space\space \cdots\space\space\space \hat{\beta_p} \space\space]^T$ and $\hat{\beta_0}=\bar{Y}$

now, even though method 1 seems more natural, most books follow method 2. why so? What advantage does method 2 have over method 1?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.