# How come regularization knows which coefficients to decrease and which coefficients to increase/leave as they are?

I am just learning about regularization and I haven't found a satisfying answer to this question. When I perform regularization, how come the right coefficients get decreased or turned into $$0$$?

I just learned that for ridge regression we add the term:

$$\lambda \sum_{j=1}^p \beta_j^2$$

to the cost function. $$\lambda$$ is the regularization parameter, $$p$$ is the number of predictors and $$\beta_j$$ are the coefficients of the model's parameters.

I learned that in this type of regression the coefficients can be decreased, but they are never quite $$0$$.

In the case of lasso regression we add the term:

$$\lambda \sum_{j=1}^p |\beta_j|$$

to the cost function. Unlike ridge regression, here the coefficients can be turned to $$0$$.

So far so good. What really perplexes me, seems like magic, is how come the right coefficients are decreased/turned into $$0$$, and the right coefficients are increased/left out as they are? How come regression doesn't decrease a coefficient that would actually improve the model and that we would actually consider to be important? How come it doesn't make a mistake like that? I mean, we just added a term to the cost function. It seems too simple to work, to do the right thing. How does regularization know which coefficients to mess with and which coefficients not to? I don't see anything in the math of regression that tells this algorithm which coefficients to decrease. How does it know?

• The answer is in the rest of the cost function. You're still trying to fit a function to the data by minimizing the sum of the errors (typically) of some sort. So the regularization keeps the coefficients from getting too big (and hence overfitting), while the sum-of-errors term tries to match the data more closely. Commented Apr 21, 2020 at 1:37
• I would argue that it doesn’t get it right. By regularizing, you bias the parameter estimates. However, it gets it wrong in a useful way by constraining the estimated values and keeping them from fluctuating too wildly, exchanging unbiasedness somewhat for a large reduction in variance. (Or at least we hope it gets it wrong in a useful way.)
– Dave
Commented Apr 21, 2020 at 3:57

It basically depends on how you're updating the value of your parameters. Let's first clarify what is happening without regularization. Suppose you want to minimize a function (objective function) that depends on a parameter $$p$$ and a variable $$x$$ as follows

\begin{align} f(x,p) \end{align}

and that at each step you're updating the value of your parameter $$b$$ with the next formula

\begin{align*} p = p - \gamma \nabla_{p}f(x,p) \:\:\: (1) \end{align*}

where $$\gamma$$ is normally known as the learning rate and $$\nabla$$ denotes the gradient of the function with respect to $$p$$. Now, if add a term that depends on the parameter $$p$$ to the objective function it would affect how I update my parameter $$p$$. For instance,

\begin{align} f(x,p) + \lambda p^2 \end{align}

where we're considering $$\lambda > 0$$. Thus, we reformulate our updating parameter formula as

\begin{align} p = p - \gamma [\nabla_{p}f(x,p) + 2\lambda p] \end{align}

which is equivalent to

\begin{align} p = (1) -2\gamma\lambda p \:\:\: (2) \end{align}

It is clear to see that (2) < (1). Thus, that's how regularization shrinks coefficients.

1. When you are adding the regularization term in the loss function, you are essentially penalizing the coefficients which are causing the loss to be high.
2. Now through the gradient descent, the model is incentivized to penalize all these coefficients to lower values which in turn reduces the loss.

Hope this helps building some intuition!

Basketball players cost money. To save money, it's best for a team not to have any players. However, we also want to win games.

A penalty is like penalizing the money we need to spend on players. Winning games is like maximizing likelihood (or minimizing cost).

If we want to both win games and also minimize our spending, we only keep the best players.

A penalty (which is analogous to a penalty on how much we spend) is put on a likelihood (aka cost, which is negative log likelihood usually, and is analogous to winning games)

$$Likelihood + weight*penalty$$

Define "important" as important to maximize cost/likelihood. Any penalization of the coefficients on the important covariates decreases likelihood; the answer is that with a big enough weight on the penalty, it will penalize the coefficients for the important covariates. If weight is small enough, it will try to still maximize likelihood, which can only be achieved by penalizing the non-important coefficients and leaving the important ones alone.