L2 regularization: large value of coefficient <=> large variance <=> overfitting For the loss function with L2 regularization:
$$Loss\ function + \lambda||w||^2_2.$$
I think following three things are equivalent with large probability:
large estimation value of $w^i$ <=> large estimation variance of $w^i$ <=> large probability of overfitting.
Is there any intuitive way to explain above conclusion?
One way is from the Bayesian's point, L2  regularization corresponds the $(0,\sigma^2)$-normal priority distribution of $w,$ to appear the large value, we need the large variance $\sigma^2.$ I cannot guarantee the statement is correct and definitely is not intuitive.
 A: To see why the equivalences do not hold, consider the following examples.

*

*First, a simple least-squares loss function with L2 regularization:  $y = w x + e$. Now, multiply $x$ by $10^{-8}$.  A little thought should convince us that, correspondingly, all the $w$ will be multiplied by $10^8$, giving us exactly the same $\hat{y}$ and exactly the same residuals.  The probability of overfitting is evidently exactly the same as it was before, but all the estimation values of $w$ will be $10^8 \times$ as large as before. Since we can make that $10^8$ as large a number as we want, it is evident that large estimation values of $w$ do not imply anything about the probability of overfitting - regardless of how that is defined (but see point 3 below.)


*Consider the same model, but with $\sigma^2_e = 0$.  Clearly, there will be no estimation error regardless of how large $||w||$ is.  Hand-waving over the obvious smoothness argument, we can conclude that by making $\sigma^2_e$ arbitrarily small, we can make the estimation error of $w$ arbitrarily small, regardless of how large $w$ is, and similarly for making $\sigma^2_e$ arbitrarily large.   Therefore, large (or small) $||w||$ by itself is not sufficient for us to draw any conclusions about how large the estimation error of $w$ is.


*It is not clear what you mean by "large probability of overfitting", which would require at the least a mathematical definition of "overfitting".  If you just mean "the parameter estimates $\neq$ the true values", for most real-world models the probability of this occurring is $1$.  (I am not sure I've ever run into a situation where it isn't $1$ in my practice.) Consequently, it is independent of both the magnitude and variance of the parameter estimates.
