I have come across two different ways of obtaining the ridge regression solution, which are as follows:
Method1:-(obtained from here)
$RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\Omega\beta$
In this case,
- $X$ and $Y$ need not be centered
- $X$ contains the column of ones. i.e. $X$ has dimensions $N\times (p+1)$, where $N$ is the number of data points and $p$ is the dimension of each data point.
- $$\beta=[ \space\space\beta_0 \space\space\space \beta_1 \space\space\space \beta_2 \space\space\space \cdots\space\space\space \beta_p\space\space]^T \text{ and } \Omega=\text{diag}(0,1,1,\cdots ,1)\implies \beta^T\Omega\beta=\sum_{i=1}^p\beta_i^2$$
$\implies RSS(\beta)=Y^TY-2Y^TX\beta+\beta^T(X^TX+\lambda\Omega)\beta$
$\implies \dfrac{\partial}{\partial\beta}RSS(\beta)=0\implies\hat{\beta}_{ridge}=(X^TX+\lambda\Omega)^{-1}X^TY=[ \space\space\hat{\beta_0} \space\space\space \hat{\beta_1} \space\space\space \hat{\beta_2} \space\space\space \cdots\space\space\space \hat{\beta_p} \space\space]^T$
Method2:-
$RSS(\beta) = (Y-X\beta)^T\cdot(Y-X\beta)+\lambda\beta^T\beta$
In this case,
- $X$ and $Y$ need to be centered
- $X$ does not contains the column of ones. i.e. $X$ has dimensions $N\times p$, where $N$ is the number of data points and $p$ is the dimension of each data point.
- $$\beta=[ \space\space \beta_1 \space\space\space \beta_2 \space\space\space \cdots\space\space\space \beta_p\space\space]^T \implies \beta^T\beta=\sum_{i=1}^p\beta_i^2$$
$\implies RSS(\beta)=Y^TY-2Y^TX\beta+\beta^T(X^TX+\lambda I)\beta$
$\implies \dfrac{\partial}{\partial\beta}RSS(\beta)=0\implies\hat{\beta}_{ridge}=(X^TX+\lambda I)^{-1}X^TY=[ \space\space \hat{\beta_1} \space\space\space \hat{\beta_2} \space\space\space \cdots\space\space\space \hat{\beta_p} \space\space]^T$ and $\hat{\beta_0}=\bar{Y}$
now, even though method 1 seems more natural, most books follow method 2. why so? What advantage does method 2 have over method 1?