13
$\begingroup$

In linear least squares we are trying to fit a line to data. A line is the simplest model that can be fit to the data. How is it possible for a linear model to over-fit the data? In short why do we need regularization for fitting a line to a data?

I understand that regularization improves generalization performance ie over unseen datasets and reduces over-fitting to training data. But how would regularization reduce the over-fitting issue for a least complex model (i.e linear model)?

This question was posted on stackoverflow, was redirected here. Also asked on DataScience stackexchange, but I am yet to get any answers/comments on the same.

$\endgroup$
7
  • 7
    $\begingroup$ The least complex model would be just an intercept (i.e., one parameter). A linear regression model has two parameters (slope and intercept) and is not the simplest possible. $\endgroup$ Aug 24, 2022 at 6:15
  • $\begingroup$ @GordonSmyth: In that case does regularization tend to move linear regression towards a zero slope and constant intercept model? $\endgroup$
    – user27665
    Aug 24, 2022 at 7:16
  • 2
    $\begingroup$ Yes, that would be the usual use or regularization for simple linear regression. In some special circumstances, regularizing the intercept towards zero might also make sense, but that would be unusual. $\endgroup$ Aug 24, 2022 at 8:04
  • 1
    $\begingroup$ A linear model is quite simple, but it is not the "simplest possible". A simpler model is just to predict a constant term; no slopes. $\endgroup$
    – Galen
    Aug 24, 2022 at 15:33
  • 1
    $\begingroup$ See the illustrations in my post at stats.stackexchange.com/a/354256/919 for a concrete example of what's going on. The point is that you are confusing two different senses of "linear." For more about that distinction, see stats.stackexchange.com/a/148713/919. $\endgroup$
    – whuber
    Aug 24, 2022 at 17:47

4 Answers 4

18
$\begingroup$

Great question! The need for regularization always depends on your sample size.

Imagine you do not have a lot of data, just three samples. I plotted three possible linear regression lines. The red one does not use regularization; for the green one, regularization on the slope parameter is used, and the blue regression has very strong regularization on the slope parameter.

enter image description here

Which one is the best model? We don't know. That will only become clear as more data is collected. But regularized models produce more conservative estimates, which often work better in practice.

That said, if you really only have the case of a simple linear regression (one input variable, like in my example), you will often have enough data to use an unregularized model.

But keep in mind that linear regression also works for multiple input variables. In that case, the model will have $p+1$ parameters if you have $p$ features. You need regularization if $p$ is large compared to your sample size.

In summary, a model is never a priori complex or simple, it always depends on the data you want to use the model for.

$\endgroup$
4
  • 1
    $\begingroup$ What if your three data points where exactly on the red line? Why regularization would penalize those weights (for simple regression this means that maybe w=10000) more than if they were on blue or green line? Regularization will shrink w towards 0 but it seems that it does it only when w is too high. I mean if the true model is y=1x and we try to approximate it then we don't get then same penalty when the true model is y=10000000x and we try to approximate it (given the same value of regularization). $\endgroup$
    – ado sar
    Oct 28, 2022 at 19:03
  • $\begingroup$ @adosar You're absolutely correct. But extreme values like w=10000 just happen to be rare in systems we study. See it in a Bayesian way: The regularization term represents the prior knowledge, that the true model is simple (w is small). Given enough evidence/data, however, your model will converge to the true model. But extraordinary claims (w=extremly large) demand extraordinary evidence :) $\endgroup$
    – PascalIv
    Nov 1, 2022 at 9:57
  • $\begingroup$ I can't get why large weights correspond to complex models. There is no reason that because w1 >> w2, y=w1*x is more complex than y=w2*x. The only way I can understand regularization is that large values of it will lead the model to predict the mean of the training data. $\endgroup$
    – ado sar
    Nov 1, 2022 at 16:57
  • $\begingroup$ If you have a small w1, replacing the model with predicting the training mean does not introduce a large error. It is therefore closer to the simpler model of just using the mean. I think this all becomes more clear when looking at models with more than one parameter. If you regularize a linear model with many parameters using L2 regularization, it is less able to overfit complex noise patterns. The model is simpler, but parameters usually don't shrink to zero, so the number of parameters stays the same. $\endgroup$
    – PascalIv
    Nov 2, 2022 at 8:24
10
$\begingroup$

Another way of looking at it: A linear model is not the simplest model that can be fitted to the data - a linear model with a constraint on the values of the weights is even more simple. For example, if we constrain the number of non-zero weights, we have a model that is structurally less complex - it has fewer parameters. As we increase the number of non-zero weights, we create a sequence of nested models of increasing complexity. A model with three non-zero weights can do everything a model with two non-zero weights can do, and also a few things it can't, so it must be a more complex model in some sense.

We can do something similar with regularisation, but it is a bit more subtle. We can create another sequence of models of increasing complexity by putting a constraint on the norm of the weight vector (limiting the magnitude of the weights rather than the number of non-zero weights). We then have an optimisation problem of the form:

$\mathrm{min}_\vec{\theta} f(\vec{\theta}) \quad \mathrm{s.t.} \quad \|\vec{\theta}\|^2 < C$

where $\vec{\theta}$ is the vector of model parameters (weights) and $C$ is the hyper-parameter controlling the maximum allowed value of the norm of the parameter vector and $f(\cdot)$ is the loss function (e.g. sum of squares). If we have a model with a particular value of $C$ and then increase $C$ a bit (to $C'$), then it can do anything it previously could, but it can also realize some additional mappings that it couldn't before. So as we increase $C$, we create models that are potentially more and more complex.

How does this relate to regularisation? One way of solving a constrained optimisation problem is to take the Lagrangian to turn it into an unconstrained optimisation problem, and in this case, the Lagrangian is:

$\Lambda(\vec{\theta},\lambda) = f(\vec{\theta}) + \lambda\|\vec{\theta}\|^2 - \lambda C.$

We can ignore the last term as it doesn't depend on the parameter vector, and what is left is a regularised loss function.

So we see that regularisation is a way of controlling the complexity of even a linear model, by limiting the set of mappings it can implement.

If we think of one-dimensional regression tasks, it is difficult to find a lot of value in regularisation, but when we have models with lots of parameters, regularisation becomes more obviously useful. Some of the attributes may be correlated with the target by random chance and have no predictive value. Regularisation will help surprress those attributes, and result in a model with better generalsiation.

Modelling data generally involved fitting the complexity of the model (class) to the complexity of the data we have available, and regularisation provides a convenient way of doing that (it is a continuous hyper-parameter, so it is a bit less of a blunt instrument that e.g. feature selection).

Just to add, I've been a bit vague with terminology here. Really is is nested sets of model/hypothesis classes rather than models, i.e. the set of models that could be realised within the constraint imposed by the value of the regularisation parameter (c.f. @Ben's answer).

$\endgroup$
6
$\begingroup$

I think part of the misunderstanding may be driven by the meaning of "model". A linear model is a set of distributions, where (for simplicity) we can consider that each distribution is represented by a line. Thus a linear model is a set (or collection) of lines - not a single line. The bigger that set is, the more complex the model is. Regularization--like removing variables from a regression--reduces the complexity of the model by making the model contain fewer lines. This can be helpful since more complex models can be more prone to overfitting.

$\endgroup$
5
  • 1
    $\begingroup$ In what sense, exactly, does one "represent" a distribution by a line? The sense of this is obscure and difficult to match up to standard accounts of linear models. $\endgroup$
    – whuber
    Aug 24, 2022 at 17:49
  • $\begingroup$ That helps, but the lack of clarity in your explanation persists. Moreover, what you describe is one specific linear model. In general the occurrence and meaning of "lines" is obscure: perhaps you're trying to describe a model linear in its non-nuisance parameters, but it's hard to tell. Finally, you describe a collection of conditional distributions in your comment. That differs (dramatically) from the full "distribution" referenced in your answer. These are some of the reasons I feel your answer would benefit from some clarifying edits. $\endgroup$
    – whuber
    Aug 25, 2022 at 13:32
  • $\begingroup$ Ben, my principal problem is that your vague description is likely to be misunderstood by other readers. $\endgroup$
    – whuber
    Aug 25, 2022 at 14:10
  • $\begingroup$ It is a pity you deleted your answer here. I think it was an interesting and useful perspective. The decision is of course up to you, but I found the answer helpful. And if I may reiterate: I think I have only ever seen OLS used as an estimator in a linear model. Have you seen it used for any other purpose such as an estimator in a nonlinear model or as something else than an estimator? $\endgroup$ Dec 11, 2022 at 10:44
  • $\begingroup$ All right, thank you! $\endgroup$ Dec 11, 2022 at 20:34
1
$\begingroup$

There is an elegant theoretical reason one might want to regularize a linear model. It is related to Dikran's answer, in that we are expressing an assumption about the weights. In essence, L2 regularization applied to a least squares linear fit expresses a Gaussian prior assumption on weight space. I'll show below the broad strokes of M-estimators used to derive two things:

  • (MLE) Assume Gaussian distributed observation noise $\implies$ least squares loss gives maximum likelihood model.
  • (MAP) Assume Gaussian distribution of model weights $\implies$ L2 regularization on loss gives maximum likelihood model.

I'll leave out details for brevity, since they are available broadly already. The point is that MSE loss and L2 regularization can be derived from first principles and simple distributional assumptions.

MLE from observation noise

In the linear regression setting, we learn model weights $\mathbf{w}$ to make scalar predictions $\hat{y}$ from samples $\mathbf{x}$ as

$$ \hat{y} = \mathbf{w}^T\mathbf{x} $$

When one assumes the true underlying distribution is a linear combination and a Gaussian noise term,

$$ y|\mathbf{x} = \mathbf{w}^T \mathbf{x} + \mathcal{N}(0, \sigma^2) $$

then maximum likelihood estimation (MLE) induces a mean squared error loss

$$ \mathcal{L}_{MLE}(\mathbf{w}) = \sum_{i=1}^n (\mathbf{w}^T\mathbf{x}_i - y)^2 $$

such that minimizing $\mathcal{L}_{MLE}$ produces the MLE estimate of weights.

MAP from weight distribution

Further, if one assumes a Gaussian prior distribution on the model weights $\mathbf{w}$ with each weight $w_i$ having identical variance $\nu^2$

$$ w_i \sim \mathcal{N}(0, \nu^2) $$

then the analogous maximum a posteriori (MAP) estimation induces the L2 regularizer with regularization weight $\lambda = \frac{\sigma^2}{\nu^2}$

$$ \mathcal{L}_{MAP}(\mathbf{w}) = \sum_{i=1}^n (\mathbf{w}^T\mathbf{x}_i - y)^2 + \lambda||\mathbf{w}||^2_2 $$

such that minimizing $\mathcal{L}_{MAP}$ produces the MAP estimate of weights.

So choosing least squares loss expresses a Gaussian observation noise assumption. And choosing L2 regularization expresses a Gaussian model weight assumption, where $\lambda$ expresses an assumed variance ratio between observation noise and model weights.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.