Given this equation -: ∆(l) = ∆(l) + δ(l+1)(a(l))T

How can we calculate ∆(L) as this would require calculating δ(L+1) which never exists in the first place. I am stuck with Back Propagation implementation, though I have gone through the material multiple number of times. If I understood the formula right , then for calculating the partial derivative of a layer l we need to have the δ vector for the next layer. Please revert.


I think the equation belongs to the backward algorithm in backpropagation. The backward algo begins once the forward algo is done. The error computed is propagated back in backward algo. The output layer neuron is referred as i+1 whereas the current layer neuron as i.

Hope this throws you some light.

  • $\begingroup$ So it means we don't have to calculate the Δ(l)i,j for the output layer. If we do not have to calculate the partial derivative w.r.t to the last layer then in that case it makes sense, but if we have to calculate the partial derivative w.r.t to the last layer then the δ does not exists as there is nothing beyond the output layer. For eg, a nn of 1 i/p layer with 4 units, 1 hidden layer with 3 unit & 1 output layer with 2 unit then the final partial derivate vector results in vector of dimension 3 assuming final partial derivate wrt to units of i/p and o/p layer are not calculated. Right? $\endgroup$ – Nitin Misra Sep 14 '17 at 4:15

As the word implies, backpropagation starts from right to left and finishes from left to right.

At the start of the backwards process (at the output layer), the very first delta is calculated as a function of the prediction error. Specifically, for a single hidden layer MLP, the delta value of output layer neuron (regression) with predicted output $\hat y$ and actual output $y$ is calculated as:

$\delta^{[out]} = \hat{y}(1-\hat{y})(y-\hat{y}) $

This formula derives from and approximates $\nabla^{[out]} \mathbf{w} $.

The process continues from right to left and computes "delta" values for nodes at every layer $t$ using the values from the past layer $t+1$. The delta value for hidden layer neuron $i$, with activated charge $s_i = g(.)$, is calculated as:

$\delta^{[hid]}_i = g(s_i^{[hid]})(1-g(s_i^{[hid]}))({w^{[hid\rightarrow out]}_i} \cdot \delta^{[out]}) , \forall i \in {1,...,n^{[hid]}}$

This formula approximates $\nabla^{[hid]} \mathbf{w} $.

Finally, during the forward process, all network weights are recalculated as a function of the newly computed delta values.

Weight update between input and hidden layers:

$ {w^{[in\rightarrow hid]}_i} = {w^{[in\rightarrow hid]}_i} + {x^{[in]}_i} \cdot r \cdot \delta^{[hid]}_i$

Weight update between hidden and output layers:

$ {w^{[hid\rightarrow out]}_i} = {w^{[hid\rightarrow out]}_i} + {s^{[hid]}_i} \cdot r \cdot \delta^{[out]}$

for constant learning rate $r$.


Not the answer you're looking for? Browse other questions tagged or ask your own question.