# How does regularization reduce overfitting for a linear decision boundary (logistic regression)?

I understand that for higher-order polynomials, reducing the weights of individual features can help to avoid complex functions that are overfit to the training data in a logistic regression classifier.

But I'm not entirely sure how this is the case for non-polynomial features (e.g. there are no $$x_1^2$$, $$x_1\cdot x_2$$, etc. terms where $$x_1$$, $$x_2$$, etc. are features inherent to some original dataset). In other words, why is it that when I lower or completely remove the regularization term $$\sum_j\theta_j^2$$, my classifier will be more sensitive to outliers?

For instance, decision boundary of an SVM with $$C = 1000$$ (more regularization):

And an SVM with $$C = 1$$ (less regularization):

I think the reason for my confusion is that I can fully imagine a dataset that would be well fit by something like a quadratic function but where the feature space is very large and so it can be easy to overfit the dataset with a complex higher order function. I can't visualize anything of the sort for the linear case.

• I know what you mean about explanations about regularization tending to use polynomial examples. However, consider it like this: unconstrained, the parameter estimates can latch onto whatever coincidence they see, but the constraints prevent them from being able to latch on too hard and overfit. – Dave Dec 30 '20 at 3:24
• – Ben Reiniger Dec 30 '20 at 3:27

• This answer and the comments are all reasonable explanations, thanks all. I think it's starting to click, but I'm still not entirely sure how this explains a linear model's weights. For a polynomial model, I can see the effect of lowering all weights to be simplifying say, a cubic function to a quadratic or linear one because for instance the $x^2$ or $x^3$ is not as easily able to dominate. If the model is already linear, noisy data/outliers would be fit by weight vectors that are different but not necessarily smaller (in norm / length). So what is the effect of a regularization term? – khajiit Dec 30 '20 at 4:03
• @khajiit regularization is not just about the form of the model but about how many dimensions to include. The simplest model would fit $y_i=\bar{y}$ or something like that. A linear model with 1000 features is more prone to overfit than one with just 10 feature since there are fewer levers to tweak to reduce the error. – Bey Dec 30 '20 at 4:12
• Thanks for the clarification. In the case of a dataset with only two features $x_1$ and $x_2$ (I added pictures to my post above for clarity), how does this lever-tweaking analogy hold? Two features seems like a relatively small amount with or without regularization in terms of addressing that outlier. Apologies for the extra requests for clarification, this has already been helpful in any event. – khajiit Dec 30 '20 at 4:25