This question already has an answer here:

Why increasing lambda parameter in L2-regularization makes the co-efficient values converge to zero?

I have just tried to do the math, but it's a little bit rusted.

Lets say that we have a simple linear model as follows: $y=w_1\cdot x$

we could write the cost function for ridge regression is to be minimized:

$cost(\hat{w_1}, \lambda)= (y - \hat{w_1} \cdot x)^2 + \lambda \cdot \hat{w_1}^2$

it means that if we consider the problem as min-max:

$\frac{\hat{dw_1}}{dc} = -2 \cdot x \cdot (y - \hat{w_1}) + 2\cdot \lambda \cdot \hat{w_1} = 0$ so,

$y = (1 + \frac{\lambda}{x}) \cdot \hat{w_1}$

Since the y and x are invariants, it is to be expected increasing $\lambda$ make the co-efficient decrease as the equation holds.

Is that the right way to reason?


marked as duplicate by Michael Chernick, kjetil b halvorsen, mkt, jpmuc, Ferdi Jan 28 at 7:47

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ Apply the formula given in the question (and answers) at stats.stackexchange.com/questions/69205/…. Find out more by searching our site for "ridge regression." $\endgroup$ – whuber Jan 22 at 21:23
  • 1
    $\begingroup$ Yes, you are reasoning is fine. $\endgroup$ – usεr11852 Jan 22 at 21:28

Yep, that is one way to think about it, although it seems a tad obscure to me.

I think it's simpler to just look at your $\text{cost}$ equation:

$\text{cost}(\hat{w_1}, \lambda) = (y - \hat{w_1} \cdot x)^2 + \lambda \cdot \hat{w_1}^2$

We can see from this that, for large $\lambda$, our cost increases quadratically with the absolute size of $\hat{w_1}$. That is, we are penalising our model for having a large weight: thus to reduce the cost, our $\hat{w_1}$ coefficient is shrunk towards zero.

If $\lambda$ is small, or zero, this second term doesn't really affect the cost, so $\hat{w_1}$ is free to grow as large as it needs to, to minimise the other component of the cost function.


Not the answer you're looking for? Browse other questions tagged or ask your own question.