In the $\delta$-rule which is used for error back propagation in neural networks, there is a factor $f'\!\left(\sum_i w_{ij}y_i\right)$, often written as $f'(\text{net}_i)$, which is just the derivative of the activation or transfer function evaluated at the weighted sum of the input. The derivative of $\tanh$ is bell-shaped so that values $\left|x\right|\geqslant3$ it is almost zero:


This means that units with a strong activation in either way have a more limited scope of action, where as values close to zero have more leeway. If I'm understanding it correctly, it keeps neurons that have already strongly decided one way or the other in their configuration. What is the purpose of this for weight updating? How would the networks behave differently if one would remove or change this factor?

As a reminder, the full $\delta$-rule looks like this for hidden neurons and output neurons:

$$\Delta w_{ij} = \eta\;\delta_j\;y_i$$

$$\delta_j = (\hat y_j - y_j)\;f'(\text{net}_j) \quad \text{or} \quad \delta_j = \sum\limits_k^K\left(w_{jk}\delta_k\right)\;f'(\text{net}_j)$$

where $\Delta w_{ij}$ is the change of the weight that connects the neuron $i^{\text{th}}$ neuron in the preceding layer with the $j^{\text{th}}$ neuron in the layer we are currently considering; $\eta$ is the learning rate, $y_j$ is the output of the current neuron, $\hat y_j$ is the teacher value for the $j^{\text{th}}$ output, $y_i$ is the output of the $i^{\text{th}}$, and $k$ is an index in the subsequent layer.


1 Answer 1


I think that you are overanalysing this a bit too much. I am sure you are aware that the back-propagation algorithm is mathematically derived by search for the derivates of the weights with respect to the loss.

Therefore the derivative of the activation function evaluated at the net input of the layer is a necessaty in the backward-pass and can not be removed or changed. I mean you can obviously remove/alter this aspect, but the mathematical foundation of back-prop will become invalid which would result in ineffective weight updates (i.e. you are no longer moving in the direction of the gradient) and thus is likely to yield worse convergence and/or a worse optima when training. What exactly would happen can not be determined, unless you start fixing other aspects of the network and do an empirical comparison.

I do like your analysis on the bell-shape of the derivative of the activation function (which btw a sigmoid activation function also has), however don't get tricked into over analysing this, as for example a relu activation function (max(0,x)), which is often used, has a constant derivative. So in fact you could argue that you can remove this property by switching to a relu unit although you would have to adjust the forward pass as well of course

At last if you are looking for a particular problem with the tanh, sigmoid activation functions then you should focus on the fact that they are saturating non-linearities. This makes it such that large layer inputs result in similar activations even if they are quite far apart. This indifference to change when the net input is large (or very small) may trouble training, as in the forward and backward pass the changes are unnoticeable.

  • $\begingroup$ Isn't the direction of the gradient determined by the error differentials which are passed from the subsequent layer or the teacher-output difference? The co-domain of the derivative of $\tanh$ is positive, so it can't have an effect of the direction of the gradient unless I'm overlooking something. $\endgroup$
    – Lenar Hoyt
    Nov 12, 2015 at 0:57
  • $\begingroup$ That is correct. The initial direction is determined by the direction of the deltas in the last layer. This direction is then back-propagated through each layer, where the weights alter the direction according to the contribution of the layer in the forward pass $\endgroup$
    – Sjoerd
    Nov 12, 2015 at 9:27
  • 1
    $\begingroup$ Oh, I see what you are trying to say. Indeed when you would remove, or alter the derivative term then this would not lead to a different gradient direction, however the magnitude will change (and at each consecutive layer this effect will be larger). It therefore may occur that you overshoot/undershoot the proposed step, from which there may be another optimum which is more attractive. I am thus saying that you will likely end up in a different optimum, be it worse or better. This makes it very unpredictable. Especially when combining this with stochastic grad desc as opposed to grad desc $\endgroup$
    – Sjoerd
    Nov 12, 2015 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.