# How to derive errors in neural network with the backpropagation algorithm?

From this video by Andrew Ng around 5:00

How are $\delta_3$ and $\delta_2$ derived? In fact, what does $\delta_3$ even mean? $\delta_4$ is got by comparing to y, no such comparison is possible for the output of a hidden layer, right?

• The video link is not working. Please, update it, or provide a link to the course. Thanks. – MadHatter Nov 28 '17 at 3:09

I'm going to answer your question about the $\delta_i^{(l)}$, but remember that your question is a sub question of a larger question which is why:

$$\nabla_{ij}^{(l)} = \sum_k \theta_{ki}^{(l+1)}\delta_k^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

Reminder about the steps in Neural networks:

• Step 1: forward propagation (calculation of the $a_{i}^{(l)}$)

• Step 2a: backward propagation: calculation of the errors $\delta_{i}^{(l)}$

• Step 2b: backward propagation: calculation of the gradient $\nabla_{ij}^{(l)}$ of J($\Theta$) using the errors $\delta_{i}^{(l+1)}$ and the $a_{i}^{(l)}$,

• Step 3: gradient descent: calculate the new $\theta_{ij}^{(l)}$ using the gradients $\nabla_{ij}^{(l)}$

First, to understand what the $\delta_i^{(l)}$ are, what they represent and why Andrew NG it talking about them, you need to understand what Andrew is actually doing at that pointand why we do all these calculations: He's calculating the gradient $\nabla_{ij}^{(l)}$ of $\theta_{ij}^{(l)}$ to be used in the Gradient descent algorithm.

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial \theta_{ij}^{(l)}}$$

As we can't really solve this formula directly, we are going to modify it using TWO MAGIC TRICKS to arrive at a formula we can actually calculate. This final usable formula is:

$$\nabla_{ij}^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

To arrive at this result, the FIRST MAGIC TRICK is that we can write the gradient $\nabla_{ij}^{(l)}$ of $\theta_{ij}^{(l)}$ using $\delta_i^{(l)}$:

$$\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$$ With $\delta_i^{(L)}$ defined (for the L index only) as:

$$\delta_i^{(L)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$

And then the SECOND MAGIC TRICK using the relation between $\delta_i^{(l)}$ and $\delta_i^{(l+1)}$, to defined the other indexes,

$$\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)}))$$

And as I said, we can finally write a formula for which we know all the terms:

$$\nabla_{ij}^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

DEMONSTRATION of the FIRST MAGIC TRICK: $\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$

We defined:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial \theta_{ij}^{(l)}}$$

The Chain rule for higher dimensions (you should REALLY read this property of the Chain rule) enables us to write:

$$\nabla_{ij}^{(l)} = \sum_k \dfrac {\partial C} {\partial z_k^{(l)}} * \dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}}$$

However , as:

$$z_k^{(l)} = \sum_m \theta_{km}^{(l)} * a_m^{(l-1)}$$

We then can write:

$$\dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}} = \dfrac {\partial}{\partial \theta_{ij}^{(l)}} \sum_m \theta_{km}^{(l)} * a_m^{(l-1)}$$

Because of the linearity of the differentiation [ (u + v)' = u'+ v'], we can write:

$$\dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}} = \sum_m\dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)}$$

with: $$if k,m \neq i,j, \ \ \dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} = 0$$

$$if k,m = i,j, \ \ \dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} = \dfrac {\partial\theta_{ij}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} = a_j^{(l-1)}$$

Then for k = i (otherwise it's clearly equal to zero):

$$\dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}} = \dfrac {\partial\theta_{ij}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} + \sum_{m \neq j}\dfrac {\partial\theta_{im}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} = a_j^{(l-1)} + 0$$

Finally, for k = i: $$\dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}} = a_j^{(l-1)}$$

As a result, we can write our first expression of the gradient $\nabla_{ij}^{(l)}$:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}} * \dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}}$$

Which is equivalent to:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}} * a_j^{(l-1)}$$

Or:

$$\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$$

DEMONSTRATION OF THE SECOND MAGIC TRICK: $\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)}))$ or:

$$\delta^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a^{(l)}(1-a^{(l)}))$$

Remember that we posed:

$$\delta^{(l)} = \dfrac {\partial C} {\partial z^{(l)}} \ \ \ and \ \ \ \delta_i^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$

Again, the Chain rule for higher dimensions enables us to write:

$$\delta_i^{(l)} = \sum_k \dfrac {\partial C} {\partial z_k^{(l+1)}} \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$$

Replacing $\dfrac {\partial C} {\partial z_k^{(l+1)}}$ by $\delta_k^{(l+1)}$, we have:

$$\delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$$

Now, let's focus on $\dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$. We have:

$$z_k^{(l+1)} = \sum_j \theta_{kj}^{(l+1)} * a_j^{(l)} = \sum_j \theta_{kj}^{(l+1)} * g(z_j^{(l)})$$

Then we derive this expression regarding $z_k^{(i)}$:

$$\dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \dfrac {\partial \sum_j \theta_{kj}^{(l)} * g(z_j^{(l)}) }{\partial z_i^{(l)}}$$

Because of the linearity of the derivation, we can write:

$$\dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \sum_j \theta_{kj}^{(l)} * \dfrac {\partial g(z_j^{(l)}) }{\partial z_i^{(l)}}$$

If j $\neq$ i, then $\dfrac {\partial \theta_{kj}^{(l)} * g(z_j^{(l)})} {\partial z_i^{(l)}} = 0$

As a consequence:

$$\dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \dfrac {\theta_{ki}^{(l)} * \partial g(z_i^{(l)}) }{\partial z_i^{(l)}}$$

And then:

$$\delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l)} * \dfrac { \partial g(z_i^{(l)}) }{\partial z_i^{(l)}}$$

As g'(z) = g(z)(1-g(z)), we have:

$$\delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l)} * g(z_i^{(l)})(1-g(z_i^{(l)})$$

And as $g(z_i^{(l)} = a_i^{(l)}$, we have:

$$\delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l+1)} * a_i^{(l)}(1-a_i^{(l)})$$

And finally, using the vectorized notation:

$$\nabla_{ij}^{(l)} = [\theta^{(l+1)^T}\delta^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)}))] * [a_j^{(l-1)}]$$

• @tmangin : Following Andrew Ng talk, we have $\delta_j^{(i)}$ is the error of node j in layer l. How did you get the definition of $\delta_j^{(i)}=\frac{\partial C}{\partial Z_j^{(l)}}$. – phuong Feb 9 '17 at 17:40
• @phuong Actually, I you're right to ask: only the $$\delta_i^{(L)}$$ with the highest "l" index L is defined as $$\delta_i^{(L)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$ Whereas the deltas with lower "l" indexes are defined by the following formula: $$\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)}))$$ – tmangin Feb 28 '17 at 14:20