# (1995) Bishop's cite on weight decay regularization

On the book "Neural networks for pattern recognition" [Bishop, 1995], in chapter 9 about regularization there is a paragraph that says:

Some heuristic justification for the weight-decay regularizer can be given as follows. We know that to produce an over-fitted mapping with regions of large curvature requires relatively large values for the weights. For small values of the weights the network mapping represented by a multi-layer perceptron is approximately linear, since the central region of a sigmoidal activation function can be approximated by a linear transformation.

Okey so I understand the first part I think. Looking at the image below (taken from Goodfellow.I, deeplearningbook.org), it is clear that for the rightmost figure, the learned function has regions of large curvature, which are explained by high values of the weights of the network.

However, I don't understand what he meant with the second part of the paragraph, namely with the sentence "For small values of the weights the network mapping represented by a multi-layer perceptron is approximately linear, since the central region of a sigmoidal activation function can be approximated by a linear transformation" .

Does he mean that, if the weights are small, then $w_i * feature_i$ will be small and will therefore (assuming sigmoid activation function) lie in the linear region of the sigmoid activation function? Because if that is what he means, then why would we use sigmoid functions? It wouldn't make sense if we are not using the non-lineariy that it provides, right? Or am I missing something?

Thanks!! Does he mean that, if the weights are small, then $w_i$∗$feature_i$ will be small and will therefore (assuming sigmoid activation function) lie in the linear region of the sigmoid activation function?

In short: yes. But, as he states, this is a more or less completely heuristic description. Or one could call it "hand-waving" and convincing.

Because if that is what he means, then why would we use sigmoid functions? It wouldn't make sense if we are not using the non-linearity that it provides, right? Or am I missing something?

The answer to this is more complicated, and a topic still under discussion. I have to admit that I can't recall directly, where I have read a more complete discussion on the choice of activation functions, but most likely it's in the mentioned book itself.

The non-linearity of the activation function is definitely an important part to make ANNs being able to approximate any function. But apart from the theoretical description, there are also practical concerns. If the data and the current state of the ANN during training makes use of the non-linear part "too much", then it can become volatile, inaccurate and slow. I'd have to think a lot more, to make a mathematical argument (or even proof). But this is also a part of the aforementioned discussion (maybe it's in "An Introduction to Statistical Learning" by James, Witten, Hastie and Tibshirani).

So -- no, you're not missing something, if you focus just on the question itself. But there is a bigger picture to it, which make the focus a lot softer.

• I read this in deeplearningbook [Goodfellow. I], when talking about the ReLU function: "Because rectified linear units are nearly linear, they preserve many of the properties that make linear models easy to optimize with gradient based methods." which may be related with what you say. But do you agree with the thought that if all the weights are pushed towards the linear part of the ReLU, then it loses a bit the sense of using a non-linear function? Or maybe using the non-linearity "a little bit" is enough? May 24, 2018 at 20:36
• @sdiabr -- this is a complex question, because it's connected to a lot of different parts. Truly linear approximations can sometimes be trivially solved (single step), in the case of a ANN this is a bit more involved, but technically still straightforward. Unfortunately the error space might be so uneven that you need a lot of elements in the NN, which makes the training complicated again. In a nutshell it is a compromise between complex training and complex description. Bishop has published several papers on this topic, including calculations on "single-step" solutions for network weights. May 25, 2018 at 12:17
• when could your sentence "unfortunately the error..." happen? I didn't get that. If using linear activation functions? May 31, 2018 at 13:42
• @sdiabr: no, the shape of the error space -- usually called "error surface" -- is a feature of your problem, or rather the data set itself. If you use a linear model, then you don't have this problem, because you are blind to it. Linear models have many nice features, especially when it comes to minimization -- e.g. gradient descent. But they lack in description power; so there is always a trade-off. May 31, 2018 at 14:17
• okey I see now, thanks. And this error surface that you mention is defined by the data itself and also the loss function that we choose to optimize right? May 31, 2018 at 18:42