1
$\begingroup$

My class started learning about ridge regression two weeks ago. Before that we learned about Lagrange multipliers and the connection between that and ridge penalty/constrain function.

Ridge:

enter image description here

Lagrange:

enter image description here

My question is why is the constant $c$ is not shown in ridge regression? Are we setting $c=0$? So is our constrain in ridge regression to set the sum of $\beta_j^2$ to zero?

Also, since $\beta_j^2$ is positive doesn't that mean that we are requiring all $\beta_j$ to equal to zero?

$\endgroup$
0
$\begingroup$

Since the penalty multiplier $\lambda$ is finite, no, we're not requiring $b_j$ to be zero. However, if we keep increasing the number of parameters, then it pushes them closer to zero, since the sum $\sum_jb_j$ is penalized. The idea of the ridge regression is to limit the magnitude of the parameters.

If you were to use the penalty as $\lambda\sum_j(b_j-c)^2$ then the implications is that $Xb$ would be more likely to overfit. By forcing $b_j$ to be small we limit the ability of $Xb$ to overfit the data.

$\endgroup$
  • $\begingroup$ I am confused as to why we don't write the penalty function as $\lambda(\sum_j(b_j)^2-c)$ since that is how Lagrange multipliers usually written. Also the solution must satisfy $\sum_j(b_j)^2=c$ because that is how we learned Lagrange multipliers. $\endgroup$ – gbd Apr 24 at 3:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.