I am just learning about regularization and I haven't found a satisfying answer to this question. When I perform regularization, how come the right coefficients get decreased or turned into $0$?

I just learned that for ridge regression we add the term:

$$\lambda \sum_{j=1}^p \beta_j^2$$

to the cost function. $\lambda$ is the regularization parameter, $p$ is the number of predictors and $\beta_j$ are the coefficients of the model's parameters.

I learned that in this type of regression the coefficients can be decreased, but they are never quite $0$.

In the case of lasso regression we add the term:

$$\lambda \sum_{j=1}^p |\beta_j|$$

to the cost function. Unlike ridge regression, here the coefficients can be turned to $0$.

So far so good. What really perplexes me, seems like magic, is how come the right coefficients are decreased/turned into $0$, and the right coefficients are increased/left out as they are? How come regression doesn't decrease a coefficient that would actually improve the model and that we would actually consider to be important? How come it doesn't make a mistake like that? I mean, we just added a term to the cost function. It seems too simple to work, to do the right thing. How does regularization know which coefficients to mess with and which coefficients not to? I don't see anything in the math of regression that tells this algorithm which coefficients to decrease. How does it know?

  • $\begingroup$ The answer is in the rest of the cost function. You're still trying to fit a function to the data by minimizing the sum of the errors (typically) of some sort. So the regularization keeps the coefficients from getting too big (and hence overfitting), while the sum-of-errors term tries to match the data more closely. $\endgroup$ – Adrian Keister Apr 21 at 1:37
  • $\begingroup$ I would argue that it doesn’t get it right. By regularizing, you bias the parameter estimates. However, it gets it wrong in a useful way by constraining the estimated values and keeping them from fluctuating too wildly, exchanging unbiasedness somewhat for a large reduction in variance. (Or at least we hope it gets it wrong in a useful way.) $\endgroup$ – Dave Apr 21 at 3:57

It basically depends on how you're updating the value of your parameters. Let's first clarify what is happening without regularization. Suppose you want to minimize a function (objective function) that depends on a parameter $p$ and a variable $x$ as follows

\begin{align} f(x,p) \end{align}

and that at each step you're updating the value of your parameter $b$ with the next formula

\begin{align*} p = p - \gamma \nabla_{p}f(x,p) \:\:\: (1) \end{align*}

where $\gamma$ is normally known as the learning rate and $\nabla$ denotes the gradient of the function with respect to $p$. Now, if add a term that depends on the parameter $p$ to the objective function it would affect how I update my parameter $p$. For instance,

\begin{align} f(x,p) + \lambda p^2 \end{align}

where we're considering $\lambda > 0$. Thus, we reformulate our updating parameter formula as

\begin{align} p = p - \gamma [\nabla_{p}f(x,p) + 2\lambda p] \end{align}

which is equivalent to

\begin{align} p = (1) -2\gamma\lambda p \:\:\: (2) \end{align}

It is clear to see that (2) < (1). Thus, that's how regularization shrinks coefficients.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy