So far I've only seen this solution: $$\beta = (X^TX+\lambda I)^{-1}X^Ty.$$
But I assume this is for the case: $$y=\beta X+ \epsilon$$
What would be solution for the more general case:
$$y=\beta X+ \beta_0 +\epsilon$$
And how can I derive it?
$\begingroup$
$\endgroup$
8
-
$\begingroup$ $$\beta = (X^TX)^{-1}X^Ty$$ is not a solution to ridge regression (except perhaps in the degenerate case that no regularization is applied). Perhaps you mean $$\beta = (X^TX + \lambda I)^{-1}X^Ty,$$ where $\lambda >0$ is some well-chosen scalar, but that's meeting you more than half-way. There is an ambiguity in the question: do you want to regularize $\beta_0$ or not? $\endgroup$– Sycorax ♦Jan 18 at 19:11
-
$\begingroup$ Yes, you're right. I copied the wrong equation. How would it be including the intercept? I do not want to regularize $\beta_0$. $\endgroup$– CaterinaJan 18 at 19:21
-
1$\begingroup$ I already wrote it down, in the second equation: append $\beta_0$ to your coefficient vector, and a column of 1s to $X$. This is just as you estimate an intercept for ordinary least squares regression. $\endgroup$– Sycorax ♦Jan 18 at 19:22
-
2$\begingroup$ Any variable you like is included in the general formulation of that other thread. I don't see any place where intercepts were excluded. $\endgroup$– whuber ♦Jan 18 at 20:25
-
1$\begingroup$ Again, this is solely a matter of how you choose to write down $\beta$ and $X$. Separating it out from the rest of the coefficients, or including it among them (with a corresponding column of 1s in $X$), is exactly the same. Let me pose it to you this way: what's the OLS estimator for $\beta, \beta_0$ when you write $y= X\beta+ \beta_0 +\epsilon$? $\endgroup$– Sycorax ♦Jan 18 at 20:58
|
Show 3 more comments
1 Answer
$\begingroup$
$\endgroup$
4
The outline of how to derive the ridge regression solution in How to derive the ridge regression solution? is complete. I think you're simply stumbling over differences in notation; some sources include a column of 1s in $X$ and an intercept in $\beta$ and others don't, instead writing $y=\beta_0 + X\beta + \epsilon$. You have to pay close attention to what you're reading.
Centering $y$ and centering the columns of $X$ produces a result where the intercept $\beta_0 =0$ exactly. This is also completely standard, because then estimating intercept can be ignored entirely, yielding the form that you are familiar with. (Thanks for Sextus for making this connection in comments.)
If this isn't the source of confusion, and you really do wish to regularize the intercept, then using $$\beta = (X^TX + \lambda I)^{-1}X^Ty$$ does the job: $X$ includes a column of 1s plus any "features" or "independent variables," and $\beta$ includes the intercept $\beta_0$.
If you do not wish to regularize the intercept (as is standard practice; see Reason for not shrinking the bias (intercept) term in regression), just define $\gamma = \begin{bmatrix}\beta_0 \\ \beta\end{bmatrix}$ and write down $$\gamma = (X^TX + \lambda A)^{-1}X^Ty,$$ where $$A= \begin{bmatrix}0 & 0 \\ 0 & I\end{bmatrix}$$ and the first column of $X$ is 1s. This "turns off" the regularization of $\beta_0$. It distinguishes between regularizing $\beta$ and applying no regularization to $\beta$.
-
$\begingroup$ Ok, it makes sense. Why don't most textbooks use this solution? I really don't see how y=0 when x=0 in most cases. I would prefer if they showed a more general solution than the degenerate case. Anyways thanks! $\endgroup$– CaterinaJan 18 at 19:29
-
4$\begingroup$ I think you're simply stumbling over differences in notation; some sources include a column of 1s in $X$ and an intercept in $\beta$ and others don't, instead writing $y=\beta_0 + X\beta + \epsilon$. You have to pay close attention to what you're reading. $\endgroup$– Sycorax ♦Jan 18 at 19:33
-
$\begingroup$ @Caterina, the reason why is maybe because many solutions will center the variables in order to get a solution where the intercept is not regularised. Then the regression on the centered data is performed without an intercept. $y= X\beta+ \epsilon$ is general because it can include an intercept, and if you don't want the intercept regularised as part of $X$ then you can use centering. $\endgroup$ Jan 18 at 22:21
-
$\begingroup$ Just another moan about ML language: in a linear model, calling the intercept term $\beta_0$ "the bias" only really makes sense if you have already centred all the data $\endgroup$– HenryJan 19 at 10:03