L2 regularization: large value of coefficient <=> large variance <=> overfitting

Question

For the loss function with L2 regularization:

$$Loss\ function + \lambda||w||^2_2.$$

I think following three things are equivalent with large probability:

large estimation value of $w^i$ <=> large estimation variance of $w^i$ <=> large probability of overfitting.

Is there any intuitive way to explain above conclusion?

One way is from the Bayesian's point, L2 regularization corresponds the $(0,\sigma^2)$-normal priority distribution of $w,$ to appear the large value, we need the large variance $\sigma^2.$ I cannot guarantee the statement is correct and definitely is not intuitive.

Answer 1

To see why the equivalences do not hold, consider the following examples.

1. First, a simple least-squares loss function with L2 regularization: $y = w x + e$. Now, multiply $x$ by $10^{-8}$. A little thought should convince us that, correspondingly, all the $w$ will be multiplied by $10^8$, giving us exactly the same $\hat{y}$ and exactly the same residuals. The probability of overfitting is evidently exactly the same as it was before, but all the estimation values of $w$ will be $10^8 \times$ as large as before. Since we can make that $10^8$ as large a number as we want, it is evident that large estimation values of $w$ do not imply anything about the probability of overfitting - regardless of how that is defined (but see point 3 below.)

2. Consider the same model, but with $\sigma^2_e = 0$. Clearly, there will be no estimation error regardless of how large $||w||$ is. Hand-waving over the obvious smoothness argument, we can conclude that by making $\sigma^2_e$ arbitrarily small, we can make the estimation error of $w$ arbitrarily small, regardless of how large $w$ is, and similarly for making $\sigma^2_e$ arbitrarily large. Therefore, large (or small) $||w||$ by itself is not sufficient for us to draw any conclusions about how large the estimation error of $w$ is.

3. It is not clear what you mean by "large probability of overfitting", which would require at the least a mathematical definition of "overfitting". If you just mean "the parameter estimates $\neq$ the true values", for most real-world models the probability of this occurring is $1$. (I am not sure I've ever run into a situation where it isn't $1$ in my practice.) Consequently, it is independent of both the magnitude and variance of the parameter estimates.