1
$\begingroup$

Quoting Prof. Bengio from his Deep Learning text (http://www.iro.umontreal.ca/~bengioy/dlbook/regularization.html),

$ w = (X^{T}X + \alpha I)^{-1}X^{T}y $

We can see L2 regularization causes the learning algorithm to “perceive” the input X as having higher variance, which makes it shrink the weights on features whose covariance with the output target is low compared to this added variance

After spending an hour, I can't understand how to approach the proof of this. Can anybody help me get an intuition for this?

$\endgroup$
4
  • $\begingroup$ $X^Ty$ is your cross product matrix (akin to $E[XY]$). $X^TX+\alpha I$ is the variability of your $x$ values, with added ridge weights. $\endgroup$
    – user75138
    Jun 29, 2015 at 16:48
  • 1
    $\begingroup$ Shouldn't we consider the entire $ (X^{T}X + \alpha I )^{-1} X^{T} $ for the proof? Can you please elaborate? $\endgroup$ Jun 29, 2015 at 16:55
  • $\begingroup$ Yes, you should, but I was offering pointers on what the compoents of the results are. $\endgroup$
    – user75138
    Jun 29, 2015 at 16:56
  • $\begingroup$ Basically $ w_{new} - w = (( X^{T}X + \alpha I)^{-1}X^{T} - ( X^{T}X)^{-1}X^{T})y $ and due to the addition of $ \alpha \delta_{ij} $, the values should be lesser in $ ( X^{T}X + \alpha I)^{-1}X^{T} $, since $ \alpha > 0 $ and the resulting matrix has greater determinant, hence lesser value. Though this indicates towards a more stronger result. Am i correct in my reasoning? $\endgroup$ Jun 29, 2015 at 18:31

1 Answer 1

2
$\begingroup$

The solution was along the similar lines of previous approach taken for the hessian approximation in the book. For getting exact relationship between $ w^{\sim} $ and $ w^{*} $, one has to do some kind of decomposition. Here, instead of eigenvalue decomposition (which is not possible, since $ X $,is not a square matrix) we do an SVD decomposition.

For unregularized version,

\begin{align} \newcommand{\new}{{\rm new}} X &= UDV^{T} \\ w &= (X^{T}X)^{-1}X^{T}y \end{align}

Putting the values of the decomposition into $w$, we get:

$$ w = D^{-1}U^{T}y $$

However if you consider the least square regularized version,

$$ w_{\new} = (X^{T}X + \alpha I)^{-1}X^{T}y $$

We get:

$$ w_{\new} = (D^2 + \lambda I)^{-1}D^{2}U^{T}y $$

We can see that $ w_{\new} $ is nothing but a scaled version of $ w $, namely $ w_{\new,j} = d_{j}^{2}/(d_{j}^{2} + \lambda_{j})w_{j} $. Hence proved.

$\endgroup$
2
  • $\begingroup$ @user3303, do not edit someone else's answer to 'correct' it. If you think it is wrong, leave a comment, downvote, or post your own (correct) answer to set the record straight. $\endgroup$ Apr 27, 2017 at 15:37
  • $\begingroup$ shouldn't it be $w_{new, j} = d_{j}/(d_{j}^{2} + \lambda_{j})w_{j} $ because $w$ contains $D^{-1}$ so it cancels out the square of $d_{j}$? $\endgroup$
    – Agile Bean
    May 11, 2021 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.