(1995) Bishop's cite on weight decay regularization

Question

On the book "Neural networks for pattern recognition" [Bishop, 1995], in chapter 9 about regularization there is a paragraph that says:

Some heuristic justification for the weight-decay regularizer can be given as follows. We know that to produce an over-fitted mapping with regions of large curvature requires relatively large values for the weights. For small values of the weights the network mapping represented by a multi-layer perceptron is approximately linear, since the central region of a sigmoidal activation function can be approximated by a linear transformation.

Okey so I understand the first part I think. Looking at the image below (taken from Goodfellow.I, deeplearningbook.org), it is clear that for the rightmost figure, the learned function has regions of large curvature, which are explained by high values of the weights of the network.

However, I don't understand what he meant with the second part of the paragraph, namely with the sentence "For small values of the weights the network mapping represented by a multi-layer perceptron is approximately linear, since the central region of a sigmoidal activation function can be approximated by a linear transformation" .

Does he mean that, if the weights are small, then $w_i * feature_i$ will be small and will therefore (assuming sigmoid activation function) lie in the linear region of the sigmoid activation function? Because if that is what he means, then why would we use sigmoid functions? It wouldn't make sense if we are not using the non-lineariy that it provides, right? Or am I missing something?

Thanks!!

Answer 1

Does he mean that, if the weights are small, then $w_i$∗$feature_i$ will be small and will therefore (assuming sigmoid activation function) lie in the linear region of the sigmoid activation function?

In short: yes. But, as he states, this is a more or less completely heuristic description. Or one could call it "hand-waving" and convincing.

Because if that is what he means, then why would we use sigmoid functions? It wouldn't make sense if we are not using the non-linearity that it provides, right? Or am I missing something?

The answer to this is more complicated, and a topic still under discussion. I have to admit that I can't recall directly, where I have read a more complete discussion on the choice of activation functions, but most likely it's in the mentioned book itself.

The non-linearity of the activation function is definitely an important part to make ANNs being able to approximate any function. But apart from the theoretical description, there are also practical concerns. If the data and the current state of the ANN during training makes use of the non-linear part "too much", then it can become volatile, inaccurate and slow. I'd have to think a lot more, to make a mathematical argument (or even proof). But this is also a part of the aforementioned discussion (maybe it's in "An Introduction to Statistical Learning" by James, Witten, Hastie and Tibshirani).

So -- no, you're not missing something, if you focus just on the question itself. But there is a bigger picture to it, which make the focus a lot softer.