2
$\begingroup$

In the book 'Pattern Recognition and Machine Learning' by Bishop (p.257 ff.) he considers a weight decay regularizer of the error function $$\hat E(w)=E(w)+\frac{\lambda}{2}w^tw$$

where $w$ is a weight vector corresponding to all of the weights in the network.

He then states that for a rescaling of weights $w\rightarrow w/a$ this regularizer is not invariant.

An invariant regularizer is given by

$$\frac{\lambda_1}{2}\sum_{w\in W_1} w^2+\frac{\lambda_2}{2}\sum_{w\in W_2} w^2$$

where we now only consider one hidden layer with its corresponding weights $W_1$ and the weights for the output layer $W_2$. In this example we only have two layers.

He writes that for a rescaling of the weights $w\rightarrow w/a$ and a simultaneous rescaling of $\lambda_1\rightarrow \sqrt{a}\lambda_1$ this regularizer is invariant. (I don't consider the second term, as it works in the same way, just look at the first sum)

However, if I do the transformation i get for the first sum $$\frac{\lambda_1\sqrt{a}}{2}\frac{1}{a^2}\sum_{w\in W_1}w^2$$

which is clearly not invariant. I cannot find my mistake.

$\endgroup$

1 Answer 1

2
$\begingroup$

Actually, you've got it correct. The actual transformations should be the following, instead of $\sqrt{\alpha}$ and $\frac{1}{\sqrt{c}}$, respectively: $$\lambda_1\rightarrow a^2\lambda_1, \ \ \ \lambda_2\rightarrow \frac{1}{c^2}\lambda_2$$ Otherwise, it yields another loss function, and the solutions change. See this (unofficial) errata.

$\endgroup$
3
  • $\begingroup$ It is strange that such a simple error is not even in his official errata... Are you 100% sure this is correct? $\endgroup$ Mar 30, 2019 at 11:36
  • 1
    $\begingroup$ I find it strange, too. But, the math is simple. $\endgroup$
    – gunes
    Mar 30, 2019 at 11:36
  • $\begingroup$ Do you by any chance know, if invariant regularizer have a special name? $\endgroup$ Mar 30, 2019 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.