How to derive errors in neural network with the backpropagation algorithm?

Question

From this video by Andrew Ng around 5:00

How are $\delta_3$ and $\delta_2$ derived? In fact, what does $\delta_3$ even mean? $\delta_4$ is got by comparing to y, no such comparison is possible for the output of a hidden layer, right?

Answer 1

I'm going to answer your question about the $\delta_i^{(l)}$, but remember that your question is a sub question of a larger question which is why:

$$\nabla_{ij}^{(l)} = \sum_k \theta_{ki}^{(l+1)}\delta_k^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

Reminder about the steps in Neural networks:

• Step 1: forward propagation (calculation of the $a_{i}^{(l)}$)

• Step 2a: backward propagation: calculation of the errors $\delta_{i}^{(l)}$

• Step 2b: backward propagation: calculation of the gradient $\nabla_{ij}^{(l)}$ of J($\Theta$) using the errors $\delta_{i}^{(l+1)}$ and the $a_{i}^{(l)}$,

• Step 3: gradient descent: calculate the new $\theta_{ij}^{(l)}$ using the gradients $\nabla_{ij}^{(l)}$

First, to understand what the $\delta_i^{(l)}$ are, what they represent and why Andrew NG it talking about them, you need to understand what Andrew is actually doing at that pointand why we do all these calculations: He's calculating the gradient $\nabla_{ij}^{(l)}$ of $\theta_{ij}^{(l)}$ to be used in the Gradient descent algorithm.

The gradient is defined as:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial \theta_{ij}^{(l)}}$$

As we can't really solve this formula directly, we are going to modify it using TWO MAGIC TRICKS to arrive at a formula we can actually calculate. This final usable formula is:

$$\nabla_{ij}^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

Note : here mapping from 1st layer to 2nd layer is notated as theta2 and so on, instead of theta1 as in andrew ng coursera

To arrive at this result, the FIRST MAGIC TRICK is that we can write the gradient $\nabla_{ij}^{(l)}$ of $\theta_{ij}^{(l)}$ using $\delta_i^{(l)}$:

$$\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$$ With $\delta_i^{(L)}$ defined (for the L index only) as:

$$ \delta_i^{(L)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$

And then the SECOND MAGIC TRICK using the relation between $\delta_i^{(l)}$ and $\delta_i^{(l+1)}$, to defined the other indexes,

$$\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) $$

And as I said, we can finally write a formula for which we know all the terms:

$$\nabla_{ij}^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$

DEMONSTRATION of the FIRST MAGIC TRICK: $\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$

We defined:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial \theta_{ij}^{(l)}}$$

Here, the Generalized chain rule provides a way to write that equality as

$$\nabla_{ij}^{(l)} = \sum_k \dfrac {\partial C} {\partial z_k^{(l)}} * \dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}}$$.

Note that this change is not obvious, but using the following intuition it becomes clearer:

The rate of change of the cost function $C$ with repect to a weighed input $z^l_j$ depends on all contributions $a^l_j$ that $z^l_j$ influences.

However, as:

$$ z_k^{(l)} = \sum_m \theta_{km}^{(l)} * a_m^{(l-1)} $$

Here: m --> unit in layer l - 1 ;
k --> unit in layer l

  i --> unit in layer *l*
  j --> unit in layer *l*-1

We then can write:

$$\dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}} = \dfrac {\partial}{\partial \theta_{ij}^{(l)}} \sum_m \theta_{km}^{(l)} * a_m^{(l-1)}$$

Because of the linearity of the differentiation [ (u + v)' = u'+ v'], we can write:

$$\dfrac {\partial z_k^{(l)}} {\partial \theta_{ij}^{(l)}} = \sum_m\dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} $$

with: $$\text{if } k,m \neq i,j, \ \ \dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} = 0 $$

$$\text{if } k,m = i,j, \ \ \dfrac {\partial\theta_{km}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_m^{(l-1)} = \dfrac {\partial\theta_{ij}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} = a_j^{(l-1)} $$

Then for $k = i$ (otherwise it's clearly equal to zero):

$$\dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}} = \dfrac {\partial\theta_{ij}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} + \sum_{m \neq j}\dfrac {\partial\theta_{im}^{(l)}} {\partial \theta_{ij}^{(l)}}* a_j^{(l-1)} = a_j^{(l-1)} + 0 $$

Finally, for $k = i$: $$\dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}} = a_j^{(l-1)}$$

As a result, we can write our first expression of the gradient $\nabla_{ij}^{(l)}$:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}} * \dfrac {\partial z_i^{(l)}} {\partial \theta_{ij}^{(l)}}$$

Which is equivalent to:

$$\nabla_{ij}^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}} * a_j^{(l-1)}$$

Or:

$$\nabla_{ij}^{(l)} = \delta_i^{(l)} * a_j^{(l-1)}$$

DEMONSTRATION OF THE SECOND MAGIC TRICK: $\delta_i^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a_i^{(l)}(1-a_i^{(l)})) $ or:

$$\delta^{(l)} = \theta^{(l+1)^T}\delta^{(l+1)}.*(a^{(l)}(1-a^{(l)})) $$

Remember that we posed:

$$ \delta^{(l)} = \dfrac {\partial C} {\partial z^{(l)}} \ \ \ and \ \ \ \delta_i^{(l)} = \dfrac {\partial C} {\partial z_i^{(l)}}$$

Again, the Chain rule for higher dimensions enables us to write:

$$ \delta_i^{(l)} = \sum_k \dfrac {\partial C} {\partial z_k^{(l+1)}} \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$$

Replacing $\dfrac {\partial C} {\partial z_k^{(l+1)}}$ by $\delta_k^{(l+1)}$, we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$$

Now, let's focus on $\dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}}$. We have:

$$ z_k^{(l+1)} = \sum_j \theta_{kj}^{(l+1)} * a_j^{(l)} = \sum_j \theta_{kj}^{(l+1)} * g(z_j^{(l)}) $$

Then we derive this expression regarding $ z_k^{(i)}$:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \dfrac {\partial \sum_j \theta_{kj}^{(l+1)} * g(z_j^{(l)}) }{\partial z_i^{(l)}} $$

Because of the linearity of the derivation, we can write:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \sum_j \theta_{kj}^{(l+1)} * \dfrac {\partial g(z_j^{(l)}) }{\partial z_i^{(l)}} $$

If $j \neq i$, then $\dfrac {\partial \theta_{kj}^{(l+1)} * g(z_j^{(l)})} {\partial z_i^{(l)}} = 0 $

As a consequence:

$$ \dfrac {\partial z_k^{(l+1)}} {\partial z_i^{(l)}} = \dfrac {\theta_{ki}^{(l+1)} * \partial g(z_i^{(l)}) }{\partial z_i^{(l)}} $$

And then:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l+1)} * \dfrac { \partial g(z_i^{(l)}) }{\partial z_i^{(l)}}$$

As $g'(z) = g(z)(1-g(z))$, we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l+1)} * g(z_i^{(l)})(1-g(z_i^{(l)}) $$

And as $g(z_i^{(l)}) = a_i^{(l)}$, we have:

$$ \delta_i^{(l)} = \sum_k \delta_k^{(l+1)} \theta_{ki}^{(l+1)} * a_i^{(l)}(1-a_i^{(l)}) $$

And finally, using the vectorized notation:

$$\nabla_{ij}^{(l)} = [\theta^{(l+1)^T}\delta^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)}))] * [a_j^{(l-1)}]$$