# Weight shrinking in linear regression by L2 regularization

Quoting Prof. Bengio from his Deep Learning text (http://www.iro.umontreal.ca/~bengioy/dlbook/regularization.html),

$w = (X^{T}X + \alpha I)^{-1}X^{T}y$

We can see L2 regularization causes the learning algorithm to “perceive” the input X as having higher variance, which makes it shrink the weights on features whose covariance with the output target is low compared to this added variance

After spending an hour, I can't understand how to approach the proof of this. Can anybody help me get an intuition for this?

• $X^Ty$ is your cross product matrix (akin to $E[XY]$). $X^TX+\alpha I$ is the variability of your $x$ values, with added ridge weights.
– user75138
Jun 29, 2015 at 16:48
• Shouldn't we consider the entire $(X^{T}X + \alpha I )^{-1} X^{T}$ for the proof? Can you please elaborate? Jun 29, 2015 at 16:55
• Yes, you should, but I was offering pointers on what the compoents of the results are.
– user75138
Jun 29, 2015 at 16:56
• Basically $w_{new} - w = (( X^{T}X + \alpha I)^{-1}X^{T} - ( X^{T}X)^{-1}X^{T})y$ and due to the addition of $\alpha \delta_{ij}$, the values should be lesser in $( X^{T}X + \alpha I)^{-1}X^{T}$, since $\alpha > 0$ and the resulting matrix has greater determinant, hence lesser value. Though this indicates towards a more stronger result. Am i correct in my reasoning? Jun 29, 2015 at 18:31

The solution was along the similar lines of previous approach taken for the hessian approximation in the book. For getting exact relationship between $w^{\sim}$ and $w^{*}$, one has to do some kind of decomposition. Here, instead of eigenvalue decomposition (which is not possible, since $X$,is not a square matrix) we do an SVD decomposition.

For unregularized version,

\begin{align} \newcommand{\new}{{\rm new}} X &= UDV^{T} \\ w &= (X^{T}X)^{-1}X^{T}y \end{align}

Putting the values of the decomposition into $w$, we get:

$$w = D^{-1}U^{T}y$$

However if you consider the least square regularized version,

$$w_{\new} = (X^{T}X + \alpha I)^{-1}X^{T}y$$

We get:

$$w_{\new} = (D^2 + \lambda I)^{-1}D^{2}U^{T}y$$

We can see that $w_{\new}$ is nothing but a scaled version of $w$, namely $w_{\new,j} = d_{j}^{2}/(d_{j}^{2} + \lambda_{j})w_{j}$. Hence proved.

• @user3303, do not edit someone else's answer to 'correct' it. If you think it is wrong, leave a comment, downvote, or post your own (correct) answer to set the record straight. Apr 27, 2017 at 15:37
• shouldn't it be $w_{new, j} = d_{j}/(d_{j}^{2} + \lambda_{j})w_{j}$ because $w$ contains $D^{-1}$ so it cancels out the square of $d_{j}$? May 11, 2021 at 7:26