2
$\begingroup$

I understand that for higher-order polynomials, reducing the weights of individual features can help to avoid complex functions that are overfit to the training data in a logistic regression classifier.

But I'm not entirely sure how this is the case for non-polynomial features (e.g. there are no $x_1^2$, $x_1\cdot x_2$, etc. terms where $x_1$, $x_2$, etc. are features inherent to some original dataset). In other words, why is it that when I lower or completely remove the regularization term $\sum_j\theta_j^2$, my classifier will be more sensitive to outliers?

For instance, decision boundary of an SVM with $C = 1000$ (more regularization): enter image description here

And an SVM with $C = 1$ (less regularization): enter image description here

I think the reason for my confusion is that I can fully imagine a dataset that would be well fit by something like a quadratic function but where the feature space is very large and so it can be easy to overfit the dataset with a complex higher order function. I can't visualize anything of the sort for the linear case.

$\endgroup$
2
  • 1
    $\begingroup$ I know what you mean about explanations about regularization tending to use polynomial examples. However, consider it like this: unconstrained, the parameter estimates can latch onto whatever coincidence they see, but the constraints prevent them from being able to latch on too hard and overfit. $\endgroup$ – Dave Dec 30 '20 at 3:24
  • 1
    $\begingroup$ See also stats.stackexchange.com/q/500360/232706 $\endgroup$ – Ben Reiniger Dec 30 '20 at 3:27
1
$\begingroup$

The idea of regularization is related to the Bayesian idea of shrinkage: you are biasing your model towards a constant fit (hence the penalty on non-zero coefficients). The regularization factor defines the "price" your model must pay to use that coefficient (i.e., make non-zero).

Therefore, if you have noisy data, the noise terms will tend to not give much reduction in RMS or whatever loss metric you have, so the model will think its better off not including them (to save on the "price" of using that coefficient).

That is how regularization reduces overfitting -- in logistic or any other linear model.

$\endgroup$
4
  • $\begingroup$ This answer and the comments are all reasonable explanations, thanks all. I think it's starting to click, but I'm still not entirely sure how this explains a linear model's weights. For a polynomial model, I can see the effect of lowering all weights to be simplifying say, a cubic function to a quadratic or linear one because for instance the $x^2$ or $x^3$ is not as easily able to dominate. If the model is already linear, noisy data/outliers would be fit by weight vectors that are different but not necessarily smaller (in norm / length). So what is the effect of a regularization term? $\endgroup$ – khajiit Dec 30 '20 at 4:03
  • $\begingroup$ @khajiit regularization is not just about the form of the model but about how many dimensions to include. The simplest model would fit $y_i=\bar{y}$ or something like that. A linear model with 1000 features is more prone to overfit than one with just 10 feature since there are fewer levers to tweak to reduce the error. $\endgroup$ – Bey Dec 30 '20 at 4:12
  • $\begingroup$ Thanks for the clarification. In the case of a dataset with only two features $x_1$ and $x_2$ (I added pictures to my post above for clarity), how does this lever-tweaking analogy hold? Two features seems like a relatively small amount with or without regularization in terms of addressing that outlier. Apologies for the extra requests for clarification, this has already been helpful in any event. $\endgroup$ – khajiit Dec 30 '20 at 4:25
  • 1
    $\begingroup$ @khajiit the C here is a bit misleading as a “regularization”. This answer gives a nice overview: datascience.stackexchange.com/questions/4943/… $\endgroup$ – Bey Dec 30 '20 at 4:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.