How exactly is the error backpropagated in backpropagation?

Question

I am reading a book on neural networks, and am now doing a chapter on backpropagation. (See chapter here). In this chapter, the writer is presenting four equations, that together form the backbone of the Backpropagation algorithm. In the second equation: \begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \end{eqnarray}

He states the following:

This equation appears complicated, but each element has a nice interpretation. Suppose we know the error: \begin{eqnarray} \delta^{l+1} \end{eqnarray} At the l+1th layer. When we apply the transpose weight matrix, \begin{eqnarray} (w^{l+1})^T \end{eqnarray} We can think intuitively of this as moving the error backward through the network, giving us some sort of measure of the error at the output of the l'th layer. We then take the Hadamard product: \begin{eqnarray} \odot \sigma'(z^l) \end{eqnarray} This moves the error backward through the activation function in layer l, giving us the error δl in the weighted input to layer l.

I don't understand how taking the transpose of a weight-matrix moves the error backwards; Since we multiply by the weight matrix to get to the next layer, some sort of division by the weight matrix would be more logical for me to get from the l+1 layer to the l'th layer. I also do not really get why taking a hadamard product takes us further back, to the weighted input. Can somebody explain me what is going on here?

Answer 1

So lets do this computation carefully in for $\delta_K^{\ell-1}$ and hopefully you'll be able to see how to build the matrix out of it:

First, we use the chain rule to write $\delta_K^{\ell-1}$ in terms of $\delta_j^{\ell}$:

$$ \delta_K^{\ell-1} = \frac{\partial C}{\partial z_k^{\ell -1}} = \sum_j \frac{\partial C}{\partial z_j^{\ell}} \frac{\partial z_j^{\ell}}{\partial z_K^{\ell-1}} = \sum_j \delta_j^{\ell}\frac{\partial z_j^{\ell}}{\partial z_K^{\ell-1}} \,. $$

Now we just need to compute the partial derivatives. Since

$$ z^{\ell}_j = \sum_{k} w_{jk}^\ell \sigma(z_k^{\ell - 1}) + b_j^\ell\,, $$ the partial derivative is $$ \frac{\partial z_j^{\ell}}{\partial z_K^{\ell-1}} = w_{jK}\sigma'(z_{K}^{\ell-1})\,. $$ Putting these pieces together we have $$ \delta_K^{\ell-1} = \sum_j \delta_j^{\ell}w_{jK}\sigma'(z_{K}^{\ell-1}) = (\delta^\ell \cdot w_{\bullet K}) \times \sigma'(z_{K}^{\ell-1})\,. $$ Here, $w_{\bullet K})$ is the $k$'th column of $w_{jk}$ as a vector. So you see that the weights appear not as an inverted quantity, but as multiples of the error. If we write this using matrix notation the transpose just takes account of the convention that $\delta^{\ell}$ is a column vector.