# What is an interpretation of the $\,f'\!\left(\sum_i w_{ij}y_i\right)$ factor in the in the $\delta$-rule in back propagation?

In the $\delta$-rule which is used for error back propagation in neural networks, there is a factor $f'\!\left(\sum_i w_{ij}y_i\right)$, often written as $f'(\text{net}_i)$, which is just the derivative of the activation or transfer function evaluated at the weighted sum of the input. The derivative of $\tanh$ is bell-shaped so that values $\left|x\right|\geqslant3$ it is almost zero:

This means that units with a strong activation in either way have a more limited scope of action, where as values close to zero have more leeway. If I'm understanding it correctly, it keeps neurons that have already strongly decided one way or the other in their configuration. What is the purpose of this for weight updating? How would the networks behave differently if one would remove or change this factor?

As a reminder, the full $\delta$-rule looks like this for hidden neurons and output neurons:

$$\Delta w_{ij} = \eta\;\delta_j\;y_i$$

$$\delta_j = (\hat y_j - y_j)\;f'(\text{net}_j) \quad \text{or} \quad \delta_j = \sum\limits_k^K\left(w_{jk}\delta_k\right)\;f'(\text{net}_j)$$

where $\Delta w_{ij}$ is the change of the weight that connects the neuron $i^{\text{th}}$ neuron in the preceding layer with the $j^{\text{th}}$ neuron in the layer we are currently considering; $\eta$ is the learning rate, $y_j$ is the output of the current neuron, $\hat y_j$ is the teacher value for the $j^{\text{th}}$ output, $y_i$ is the output of the $i^{\text{th}}$, and $k$ is an index in the subsequent layer.

• Isn't the direction of the gradient determined by the error differentials which are passed from the subsequent layer or the teacher-output difference? The co-domain of the derivative of $\tanh$ is positive, so it can't have an effect of the direction of the gradient unless I'm overlooking something. Nov 12, 2015 at 0:57