# Consisten Regularizer for Neural Network

In the book 'Pattern Recognition and Machine Learning' by Bishop (p.257 ff.) he considers a weight decay regularizer of the error function $$\hat E(w)=E(w)+\frac{\lambda}{2}w^tw$$

where $$w$$ is a weight vector corresponding to all of the weights in the network.

He then states that for a rescaling of weights $$w\rightarrow w/a$$ this regularizer is not invariant.

An invariant regularizer is given by

$$\frac{\lambda_1}{2}\sum_{w\in W_1} w^2+\frac{\lambda_2}{2}\sum_{w\in W_2} w^2$$

where we now only consider one hidden layer with its corresponding weights $$W_1$$ and the weights for the output layer $$W_2$$. In this example we only have two layers.

He writes that for a rescaling of the weights $$w\rightarrow w/a$$ and a simultaneous rescaling of $$\lambda_1\rightarrow \sqrt{a}\lambda_1$$ this regularizer is invariant. (I don't consider the second term, as it works in the same way, just look at the first sum)

However, if I do the transformation i get for the first sum $$\frac{\lambda_1\sqrt{a}}{2}\frac{1}{a^2}\sum_{w\in W_1}w^2$$

which is clearly not invariant. I cannot find my mistake.

Actually, you've got it correct. The actual transformations should be the following, instead of $$\sqrt{\alpha}$$ and $$\frac{1}{\sqrt{c}}$$, respectively: $$\lambda_1\rightarrow a^2\lambda_1, \ \ \ \lambda_2\rightarrow \frac{1}{c^2}\lambda_2$$ Otherwise, it yields another loss function, and the solutions change. See this (unofficial) errata.