So far I've only seen this solution: $$\beta = (X^TX+\lambda I)^{-1}X^Ty.$$
But I assume this is for the case: $$y=\beta X+ \epsilon$$
What would be solution for the more general case:
$$y=\beta X+ \beta_0 +\epsilon$$
And how can I derive it?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this communitySo far I've only seen this solution: $$\beta = (X^TX+\lambda I)^{-1}X^Ty.$$
But I assume this is for the case: $$y=\beta X+ \epsilon$$
What would be solution for the more general case:
$$y=\beta X+ \beta_0 +\epsilon$$
And how can I derive it?
The outline of how to derive the ridge regression solution in How to derive the ridge regression solution? is complete. I think you're simply stumbling over differences in notation; some sources include a column of 1s in $X$ and an intercept in $\beta$ and others don't, instead writing $y=\beta_0 + X\beta + \epsilon$. You have to pay close attention to what you're reading.
Centering $y$ and centering the columns of $X$ produces a result where the intercept $\beta_0 =0$ exactly. This is also completely standard, because then estimating intercept can be ignored entirely, yielding the form that you are familiar with. (Thanks for Sextus for making this connection in comments.)
If this isn't the source of confusion, and you really do wish to regularize the intercept, then using $$\beta = (X^TX + \lambda I)^{-1}X^Ty$$ does the job: $X$ includes a column of 1s plus any "features" or "independent variables," and $\beta$ includes the intercept $\beta_0$.
If you do not wish to regularize the intercept (as is standard practice; see Reason for not shrinking the bias (intercept) term in regression), just define $\gamma = \begin{bmatrix}\beta_0 \\ \beta\end{bmatrix}$ and write down $$\gamma = (X^TX + \lambda A)^{-1}X^Ty,$$ where $$A= \begin{bmatrix}0 & 0 \\ 0 & I\end{bmatrix}$$ and the first column of $X$ is 1s. This "turns off" the regularization of $\beta_0$. It distinguishes between regularizing $\beta$ and applying no regularization to $\beta$.