From this video by Andrew Ng around 5:00

enter image description here

How are $\delta_3$ and $\delta_2$ derived? In fact, what does $\delta_3$ even mean? $\delta_4$ is got by comparing to y, no such comparison is possible for the output of a hidden layer, right?

  • $\begingroup$ The video link is not working. Please, update it, or provide a link to the course. Thanks. $\endgroup$
    – MadHatter
    Nov 28, 2017 at 3:09
  • $\begingroup$ Though @tmangin answered it beautifully, still, if you want a more detailed explanation, you can refer to this GitHub repo. It is well explained and contains derivations to both the Machine Learning Course and the Deep Learning Specialization. $\endgroup$ Jun 3, 2020 at 12:50

1 Answer 1


I'm going to answer your question about the $\delta_i^{(l)}$, but remember that your question is a sub question of a larger question which is why:

$$\nabla_{ij}^{(l)} = \sum_k \theta_{ki}^{(l+1)}\delta_k^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

Reminder about the steps in Neural networks:

  • Step 1: forward propagation (calculation of the $a_{i}^{(l)}$)

  • Step 2a: backward propagation: calculation of the errors $\delta_{i}^{(l)}$

  • Step 2b: backward propagation: calculation of the gradient $\nabla_{ij}^{(l)}$ of J($\Theta$) using the errors $\delta_{i}^{(l+1)}$ and the $a_{i}^{(l)}$,

  • Step 3: gradient descent: calculate the new $\theta_{ij}^{(l)}$ using the gradients $\nabla_{ij}^{(l)}$

First, to understand what the $\delta_i^{(l)}$ are, what they represent and why Andrew NG it talking about them, you need to understand what Andrew is actually doing at that pointand why we do all these calculations: He's calculating the gradient $\nabla_{ij}^{(l)}$ of $\theta_{ij}^{(l)}$ to be used in the Gradient descent algorithm.

The gradient is defined as:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial \theta_{ij}^{(l)}}$$

As we can't really solve this formula directly, we are going to modify it using TWO MAGIC TRICKS to arrive at a formula we can actually calculate. This final usable formula is:

$$\nabla_{ij}^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

Note : here mapping from 1st layer to 2nd layer is notated as theta2 and so on, instead of theta1 as in andrew ng coursera

To arrive at this result, the FIRST MAGIC TRICK is that we can write the gradient $\nabla_{ij}^{(l)}$ of $\theta_{ij}^{(l)}$ using $\delta_i^{(l)}$:

$$\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$$ With $\delta_i^{(L)}$ defined (for the L index only) as:

$$ \delta_i^{(L)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$

And then the SECOND MAGIC TRICK using the relation between $\delta_i^{(l)}$ and $\delta_i^{(l+1)}$, to defined the other indexes,

$$\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) $$

And as I said, we can finally write a formula for which we know all the terms:

$$\nabla_{ij}^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

DEMONSTRATION of the FIRST MAGIC TRICK: $\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$

We defined:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial \theta_{ij}^{(l)}}$$

Here, the Generalized chain rule provides a way to write that equality as

$$\nabla_{ij}^{(l)} = \sum_k \dfrac {\partial C} {\partial z_k^{(l)}} * \dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}}$$.

Note that this change is not obvious, but using the following intuition it becomes clearer:

The rate of change of the cost function $C$ with repect to a weighed input $z^l_j$ depends on all contributions $a^l_j$ that $z^l_j$ influences.

However, as:

$$ z_k^{(l)} = \sum_m \theta_{km}^{(l)} * a_m^{(l-1)} $$

Here: m --> unit in layer l - 1 ;
k --> unit in layer l

  i --> unit in layer *l*
  j --> unit in layer *l*-1

We then can write:

$$\dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}} = \dfrac {\partial}{\partial \theta_{ij}^{(l)}} \sum_m \theta_{km}^{(l)} * a_m^{(l-1)}$$

Because of the linearity of the differentiation [ (u + v)' = u'+ v'], we can write:

$$\dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}} = \sum_m\dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} $$

with: $$\text{if } k,m \neq i,j, \ \ \dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} = 0 $$

$$\text{if } k,m = i,j, \ \ \dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} = \dfrac {\partial\theta_{ij}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} = a_j^{(l-1)} $$

Then for $k = i$ (otherwise it's clearly equal to zero):

$$\dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}} = \dfrac {\partial\theta_{ij}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} + \sum_{m \neq j}\dfrac {\partial\theta_{im}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} = a_j^{(l-1)} + 0 $$

Finally, for $k = i$: $$\dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}} = a_j^{(l-1)}$$

As a result, we can write our first expression of the gradient $\nabla_{ij}^{(l)}$:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}} * \dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}}$$

Which is equivalent to:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}} * a_j^{(l-1)}$$


$$\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$$

DEMONSTRATION OF THE SECOND MAGIC TRICK: $\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) $ or:

$$\delta^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a^{(l)}(1-a^{(l)})) $$

Remember that we posed:

$$ \delta^{(l)} = \dfrac {\partial C} {\partial z^{(l)}} \ \ \ and \ \ \ \delta_i^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$

Again, the Chain rule for higher dimensions enables us to write:

$$ \delta_i^{(l)} = \sum_k \dfrac {\partial C} {\partial z_k^{(l+1)}} \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$$

Replacing $\dfrac {\partial C} {\partial z_k^{(l+1)}}$ by $\delta_k^{(l+1)}$, we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$$

Now, let's focus on $\dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$. We have:

$$ z_k^{(l+1)} = \sum_j \theta_{kj}^{(l+1)} * a_j^{(l)} = \sum_j \theta_{kj}^{(l+1)} * g(z_j^{(l)}) $$

Then we derive this expression regarding $ z_k^{(i)}$:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \dfrac {\partial \sum_j \theta_{kj}^{(l+1)} * g(z_j^{(l)}) }{\partial z_i^{(l)}} $$

Because of the linearity of the derivation, we can write:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \sum_j \theta_{kj}^{(l+1)} * \dfrac {\partial g(z_j^{(l)}) }{\partial z_i^{(l)}} $$

If $j \neq i$, then $\dfrac {\partial \theta_{kj}^{(l+1)} * g(z_j^{(l)})} {\partial z_i^{(l)}} = 0 $

As a consequence:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \dfrac {\theta_{ki}^{(l+1)} * \partial g(z_i^{(l)}) }{\partial z_i^{(l)}} $$

And then:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l+1)} * \dfrac { \partial g(z_i^{(l)}) }{\partial z_i^{(l)}}$$

As $g'(z) = g(z)(1-g(z))$, we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l+1)} * g(z_i^{(l)})(1-g(z_i^{(l)}) $$

And as $g(z_i^{(l)}) = a_i^{(l)}$, we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l+1)} * a_i^{(l)}(1-a_i^{(l)}) $$

And finally, using the vectorized notation:

$$\nabla_{ij}^{(l)} = [\theta^{(l+1)^T}\delta^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)}))] * [a_j^{(l-1)}]$$

  • 1
    $\begingroup$ Thank you for your answer. I upvoted you !! Could you please cite the sources you referred for arriving at the answer... :) $\endgroup$ Jan 30, 2017 at 11:09
  • $\begingroup$ @tmangin : Following Andrew Ng talk, we have $\delta_j^{(i)}$ is the error of node j in layer l. How did you get the definition of $\delta_j^{(i)}=\frac{\partial C}{\partial Z_j^{(l)}}$. $\endgroup$
    – phuong
    Feb 9, 2017 at 17:40
  • $\begingroup$ @phuong Actually, I you're right to ask: only the $$ \delta_i^{(L)} $$ with the highest "l" index L is defined as $$ \delta_i^{(L)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$ Whereas the deltas with lower "l" indexes are defined by the following formula: $$\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) $$ $\endgroup$
    – tmangin
    Feb 28, 2017 at 14:20
  • 3
    $\begingroup$ I highly recommend reading the backprop vectorial notation of calculating the gradients. $\endgroup$
    – CKM
    Mar 26, 2017 at 13:30
  • $\begingroup$ Your final usable formula is not what Andrew Ng had, which is making it really frustrating to follow your proof. He had ∇(l)ij=θ(l)Tδ(l+1).∗(a(l)i(1−a(l)i))∗a(l−1)j, not θ(l+1)Tδ(l+1) $\endgroup$
    – azizj
    Aug 5, 2017 at 23:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.