What is an interpretation of the $\,f'\!\left(\sum_i w_{ij}y_i\right)$ factor in the in the $\delta$-rule in back propagation?

Question

In the $\delta$-rule which is used for error back propagation in neural networks, there is a factor $f'\!\left(\sum_i w_{ij}y_i\right)$, often written as $f'(\text{net}_i)$, which is just the derivative of the activation or transfer function evaluated at the weighted sum of the input. The derivative of $\tanh$ is bell-shaped so that values $\left|x\right|\geqslant3$ it is almost zero:

               

This means that units with a strong activation in either way have a more limited scope of action, where as values close to zero have more leeway. If I'm understanding it correctly, it keeps neurons that have already strongly decided one way or the other in their configuration. What is the purpose of this for weight updating? How would the networks behave differently if one would remove or change this factor?

---

As a reminder, the full $\delta$-rule looks like this for hidden neurons and output neurons:

$$\Delta w_{ij} = \eta\;\delta_j\;y_i$$

$$\delta_j = (\hat y_j - y_j)\;f'(\text{net}_j) \quad \text{or} \quad \delta_j = \sum\limits_k^K\left(w_{jk}\delta_k\right)\;f'(\text{net}_j)$$

where $\Delta w_{ij}$ is the change of the weight that connects the neuron $i^{\text{th}}$ neuron in the preceding layer with the $j^{\text{th}}$ neuron in the layer we are currently considering; $\eta$ is the learning rate, $y_j$ is the output of the current neuron, $\hat y_j$ is the teacher value for the $j^{\text{th}}$ output, $y_i$ is the output of the $i^{\text{th}}$, and $k$ is an index in the subsequent layer.

Answer 1

I think that you are overanalysing this a bit too much. I am sure you are aware that the back-propagation algorithm is mathematically derived by search for the derivates of the weights with respect to the loss.

Therefore the derivative of the activation function evaluated at the net input of the layer is a necessaty in the backward-pass and can not be removed or changed. I mean you can obviously remove/alter this aspect, but the mathematical foundation of back-prop will become invalid which would result in ineffective weight updates (i.e. you are no longer moving in the direction of the gradient) and thus is likely to yield worse convergence and/or a worse optima when training. What exactly would happen can not be determined, unless you start fixing other aspects of the network and do an empirical comparison.

I do like your analysis on the bell-shape of the derivative of the activation function (which btw a sigmoid activation function also has), however don't get tricked into over analysing this, as for example a relu activation function (max(0,x)), which is often used, has a constant derivative. So in fact you could argue that you can remove this property by switching to a relu unit although you would have to adjust the forward pass as well of course

At last if you are looking for a particular problem with the tanh, sigmoid activation functions then you should focus on the fact that they are saturating non-linearities. This makes it such that large layer inputs result in similar activations even if they are quite far apart. This indifference to change when the net input is large (or very small) may trouble training, as in the forward and backward pass the changes are unnoticeable.