Intuitive understanding of regularization

Question

As I understand from this link regularization used to reduce overfitting of model.

1. Is overfitting bad when we have really a lot of data?
2. I don't understand why "very large weights fit the training data very well"?
3. Is regularization always needed?

Answer 1

1. overfitting is always bad as it means you have done something to your model that means that it generalisation performance has become worse. This is less likely to happen when you have lots of data, and in such circumstances regularisation tends to be less helpful, but over-fitting is still something you don't want.

2. This diagram (from Wikimedia) shows an over-fitted regression model
   In order for the regression line to pass through each of the data points, the regression has high curvature at many points. At these points, the output of the model is very sensitive to changes in the value of the input variable. This generally requires model parameters of large magnitude, so that small changes in the input are magnified into large changes in the output.

3. No, regularisation is not always needed, particularly if you have so much data that the model isn't flexible enough to exploit the noise. I would recommend putting regularisation in and use cross-validation to set the regularisation parameter(s). If regularisation is unhelpful, cross-validation will tend to make the regularisation parameter small enough that it has no real effect. I tend to use leave-one-out cross-validaition as it can be computed very cheaply for many interesting models (linear regression, SVMs, kernel machines, Gaussian processes etc.), even though its high variance is less attractive.

Answer 2

1. It depends on your model & the specificity of your data. For instance, fitting an unpruned decision tree will always lead to overfitting with even just a few variables. The same goes with parametric models, where a large number of parameters can lead to overfitting, even if there is a lot of data. Eitherway, you sould systematically try to tune the complexity of your model to assure the best generalisation error.

2. I don't think it's as much a problem of "large" weights than "unconstrained" weights. Adding a regularisation term to regression basically forces your coefficients to a region near zero (or some other predefined prior value(s) ). The bayesian interpretation of regularisation makes it even more obvious : the regularisation parameter (for ridge regression, say) governs the standard deviation of the prior given to the coefficients. High regularisation means a higher "chance" for small coefficients, and constrains their value, thus reducing the freedom of your model and thus overfitting.

3. Depends on your model. If you've got huge amounts of data and 2 variables and are doing linear regression, then probably not. If you're fitting polynomials to 100 data points, then yeah. If you're unsure how complex your model is w.r.t. your training data, then just try with small amounts of regularisation and see if generalisation error improves (using a validation set or X-validation).

Answer 3

Is overfitting bad when we have really a lot of data?

Overfitting with lot of data is still overfitting and overfitting is bad.

I don't understand why "very large weights fit the training data very well"?

I found an example in Deep Learning by Goodfellow(page 293):
Suppose we apply logistic regression to a problem where the classes are linearly separable. If a set of weights $w$ make the model fit the data very well, then it's clear that $2w$ would provide us higher likelihood. And in theory after many iterations of opotimization this increasing would never hald.

Is regularization always needed?

This question seems equal to: does overfitting always occur? If there exists data the model has never seen during training, the overfitting may occur and hence regularization is necessary. It seems that hardly can we train the model with all possible cases, then normally regularization is always necessary.