I understand that for higher-order polynomials, reducing the weights of individual features can help to avoid complex functions that are overfit to the training data in a logistic regression classifier.
But I'm not entirely sure how this is the case for non-polynomial features (e.g. there are no $x_1^2$, $x_1\cdot x_2$, etc. terms where $x_1$, $x_2$, etc. are features inherent to some original dataset). In other words, why is it that when I lower or completely remove the regularization term $\sum_j\theta_j^2$, my classifier will be more sensitive to outliers?
I think the reason for my confusion is that I can fully imagine a dataset that would be well fit by something like a quadratic function but where the feature space is very large and so it can be easy to overfit the dataset with a complex higher order function. I can't visualize anything of the sort for the linear case.