# What is the geometric explanation for high variance = overfitting = bad generalization?

I am reading about the L2 regularization. According to Python Machine Learning - Second Edition, "by increasing the regularization strength via the regularization parameter λ , we shrink the weights towards zero and decrease the dependence of our model on the training data." A nice illustration for L2 regularization is given in the book: • I dont understand why the weights need to be penalised? Without the regularization term, wouldn't we be able to get to the center (optimal), which is the exact purpose of optimization of the loss function? With the regularization term, now we can not get to the lowest cost.
• Why large weights means high variance? Why large weights means the model is overfitted, or not well generalized? Can this be explained with the above figure?
• The "minimize cost" point minimizes cost for this training set. But if you sample another training set from scratch, the "minimize cost" point could be far away. Variance refers to variance of that "minimize cost" point across samples of the training set. If the variance is high, choosing that point for one training set would not generalize well to data outside the training set. So you don't actually necessarily want to get exactly to that point. You might want to reduce variance so that even though the model performs worse on this training set, it's more likely to generalize well. Jun 15, 2018 at 16:17

I do not know for which model the L2 penalization is presented. I will suppose it is linear regression, but the same discussion applies to other models, also. For other more complex models the discussion is more involved, but not impossible.

I don't understand why the weights need to be penalized?

The weights are the parameters of your model. For linear model you have $y = w^T x + \epsilon$. Since $\epsilon$ is a model for random noise, you remain only with $w$ to change in order to impose some learning bias into the model.

In linear models if the features are not independent, they geometrically look like a very closed vectors. Doing regression is the same as to find a direction in which the errors are smaller, or in other words a normal hyper plane for that direction. However this direction is not stable, because small changes in errors of features will lead to very different vectors for errors (remember they are close one to another because correlation). As a consequence it often lead to high values for weights.

Another explanation for linear models is that with correlated features we will have a matrix which is hard to be inverted, as such it leads to high values for correlation.

In general, however, having features which share a lot of information have the potential to lead to not stable models. Instability means high variance. At the same time instability means high values for weights. All those situations are covered under the statement: "small changes in input data leads to very different values in output data".

Without the regularization term, wouldn't we be able to get to the center (optimal), which is the exact purpose of optimization of the loss function?

What you said is not completely true. We have training data. Training data is just a sample of reality. Finding optimal error on sample data does not mean you can generalize. What we want to find is the optimal setup for any new data. Because of our limited sample, possible model inadequacy, possible wrong assumptions, the optimal on training data is different than optimal for any new data.

In order to search for better generalization, we need have a good compromise between bias and variance. But those to comes in opposite directions. When you decrease bias, aka go as closer as possible to optimal in training data, you start to depend on data a lot, and of course on any error or fine structure. By doing that you model become unstable. At the other side, if you decrease the variance of the model, you increase the stability and go farther from training optimal. The most stable model is the one which is independent of all features, the one which have weight 0. That model is really stable, it simply does not change no matter the input, but is useless.

You have to find a compromise, and reducing weights by regularization is one way to go.

Why large weights means high variance?

Large weights means if your input has a small error, when multiplied with large weight it become large error in output / prediction, so the model is unstable, so it has high variance.

"Why large weights means the model is overfitted, or not well generalized?"

Over fitting means you learn too much from data. That means your model learns also the irreducible error. Because it learns the noise, it have a high variance, the model is unstable. As a consequence the model, on new data, will predict using other noise (from new data) and of course will be poor as a predictor, small generalization. Since we have a high variance it means small changes in inputs leads to large changes in outputs. When this idea is applied to a function where weights multiplies or amplifies inputs in some way, one mathematical possibility is that the weights are large, most of the time.

1. The optimal is for a specific test set. The point is exactly to not get too close to it.

2. Large weights doesn't necessarily lead to high variance but you are more likely to get a large range of weight combinations for multiple training sets. Therefore, it's not doing a good job capturing the underlying pattern (if there is one).