From this video by Andrew Ng around 5:00

enter image description here

How are $\delta_3$ and $\delta_2$ derived? In fact, what does $\delta_3$ even mean? $\delta_4$ is got by comparing to y, no such comparison is possible for the output of a hidden layer, right?

  • $\begingroup$ The video link is not working. Please, update it, or provide a link to the course. Thanks. $\endgroup$ – MadHatter Nov 28 '17 at 3:09

I'm going to answer your question about the $\delta_i^{(l)}$, but remember that your question is a sub question of a larger question which is why:

$$\nabla_{ij}^{(l)} = \sum_k \theta_{ki}^{(l+1)}\delta_k^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

Reminder about the steps in Neural networks:

  • Step 1: forward propagation (calculation of the $a_{i}^{(l)}$)

  • Step 2a: backward propagation: calculation of the errors $\delta_{i}^{(l)}$

  • Step 2b: backward propagation: calculation of the gradient $\nabla_{ij}^{(l)}$ of J($\Theta$) using the errors $\delta_{i}^{(l+1)}$ and the $a_{i}^{(l)}$,

  • Step 3: gradient descent: calculate the new $\theta_{ij}^{(l)}$ using the gradients $\nabla_{ij}^{(l)}$

First, to understand what the $\delta_i^{(l)}$ are, what they represent and why Andrew NG it talking about them, you need to understand what Andrew is actually doing at that pointand why we do all these calculations: He's calculating the gradient $\nabla_{ij}^{(l)}$ of $\theta_{ij}^{(l)}$ to be used in the Gradient descent algorithm.

The gradient is defined as:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial \theta_{ij}^{(l)}}$$

As we can't really solve this formula directly, we are going to modify it using TWO MAGIC TRICKS to arrive at a formula we can actually calculate. This final usable formula is:

$$\nabla_{ij}^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

To arrive at this result, the FIRST MAGIC TRICK is that we can write the gradient $\nabla_{ij}^{(l)}$ of $\theta_{ij}^{(l)}$ using $\delta_i^{(l)}$:

$$\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$$ With $\delta_i^{(L)}$ defined (for the L index only) as:

$$ \delta_i^{(L)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$

And then the SECOND MAGIC TRICK using the relation between $\delta_i^{(l)}$ and $\delta_i^{(l+1)}$, to defined the other indexes,

$$\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) $$

And as I said, we can finally write a formula for which we know all the terms:

$$\nabla_{ij}^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

DEMONSTRATION of the FIRST MAGIC TRICK: $\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$

We defined:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial \theta_{ij}^{(l)}}$$

The Chain rule for higher dimensions (you should REALLY read this property of the Chain rule) enables us to write:

$$\nabla_{ij}^{(l)} = \sum_k \dfrac {\partial C} {\partial z_k^{(l)}} * \dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}}$$

However , as:

$$ z_k^{(l)} = \sum_m \theta_{km}^{(l)} * a_m^{(l-1)} $$

We then can write:

$$\dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}} = \dfrac {\partial}{\partial \theta_{ij}^{(l)}} \sum_m \theta_{km}^{(l)} * a_m^{(l-1)}$$

Because of the linearity of the differentiation [ (u + v)' = u'+ v'], we can write:

$$\dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}} = \sum_m\dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} $$

with: $$if k,m \neq i,j, \ \ \dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} = 0 $$

$$if k,m = i,j, \ \ \dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} = \dfrac {\partial\theta_{ij}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} = a_j^{(l-1)} $$

Then for k = i (otherwise it's clearly equal to zero):

$$\dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}} = \dfrac {\partial\theta_{ij}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} + \sum_{m \neq j}\dfrac {\partial\theta_{im}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} = a_j^{(l-1)} + 0 $$

Finally, for k = i: $$\dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}} = a_j^{(l-1)}$$

As a result, we can write our first expression of the gradient $\nabla_{ij}^{(l)}$:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}} * \dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}}$$

Which is equivalent to:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}} * a_j^{(l-1)}$$


$$\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$$

DEMONSTRATION OF THE SECOND MAGIC TRICK: $\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) $ or:

$$\delta^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a^{(l)}(1-a^{(l)})) $$

Remember that we posed:

$$ \delta^{(l)} = \dfrac {\partial C} {\partial z^{(l)}} \ \ \ and \ \ \ \delta_i^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$

Again, the Chain rule for higher dimensions enables us to write:

$$ \delta_i^{(l)} = \sum_k \dfrac {\partial C} {\partial z_k^{(l+1)}} \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$$

Replacing $\dfrac {\partial C} {\partial z_k^{(l+1)}}$ by $\delta_k^{(l+1)}$, we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$$

Now, let's focus on $\dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$. We have:

$$ z_k^{(l+1)} = \sum_j \theta_{kj}^{(l+1)} * a_j^{(l)} = \sum_j \theta_{kj}^{(l+1)} * g(z_j^{(l)}) $$

Then we derive this expression regarding $ z_k^{(i)}$:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \dfrac {\partial \sum_j \theta_{kj}^{(l)} * g(z_j^{(l)}) }{\partial z_i^{(l)}} $$

Because of the linearity of the derivation, we can write:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \sum_j \theta_{kj}^{(l)} * \dfrac {\partial g(z_j^{(l)}) }{\partial z_i^{(l)}} $$

If j $\neq$ i, then $\dfrac {\partial \theta_{kj}^{(l)} * g(z_j^{(l)})} {\partial z_i^{(l)}} = 0 $

As a consequence:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \dfrac {\theta_{ki}^{(l)} * \partial g(z_i^{(l)}) }{\partial z_i^{(l)}} $$

And then:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l)} * \dfrac { \partial g(z_i^{(l)}) }{\partial z_i^{(l)}}$$

As g'(z) = g(z)(1-g(z)), we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l)} * g(z_i^{(l)})(1-g(z_i^{(l)}) $$

And as $g(z_i^{(l)} = a_i^{(l)}$, we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l+1)} * a_i^{(l)}(1-a_i^{(l)}) $$

And finally, using the vectorized notation:

$$\nabla_{ij}^{(l)} = [\theta^{(l+1)^T}\delta^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)}))] * [a_j^{(l-1)}]$$

  • 1
    $\begingroup$ Thank you for your answer. I upvoted you !! Could you please cite the sources you referred for arriving at the answer... :) $\endgroup$ – Adithya Upadhya Jan 30 '17 at 11:09
  • $\begingroup$ @tmangin : Following Andrew Ng talk, we have $\delta_j^{(i)}$ is the error of node j in layer l. How did you get the definition of $\delta_j^{(i)}=\frac{\partial C}{\partial Z_j^{(l)}}$. $\endgroup$ – phuong Feb 9 '17 at 17:40
  • $\begingroup$ @phuong Actually, I you're right to ask: only the $$ \delta_i^{(L)} $$ with the highest "l" index L is defined as $$ \delta_i^{(L)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$ Whereas the deltas with lower "l" indexes are defined by the following formula: $$\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) $$ $\endgroup$ – tmangin Feb 28 '17 at 14:20
  • 3
    $\begingroup$ I highly recommend reading the backprop vectorial notation of calculating the gradients. $\endgroup$ – chandresh Mar 26 '17 at 13:30
  • $\begingroup$ Your final usable formula is not what Andrew Ng had, which is making it really frustrating to follow your proof. He had ∇(l)ij=θ(l)Tδ(l+1).∗(a(l)i(1−a(l)i))∗a(l−1)j, not θ(l+1)Tδ(l+1) $\endgroup$ – Aziz Javed Aug 5 '17 at 23:19

This calculation helps. The only difference from this result to Andrew's result is because of the definition of theta. In Andrew's definition, z(l+1) = theta(l)*a(l). In this calculation, z(l+1) = theta(l+1)*a(l). So actually there's no difference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.