I am reading a book on neural networks, and am now doing a chapter on backpropagation. (See chapter here). In this chapter, the writer is presenting four equations, that together form the backbone of the Backpropagation algorithm. In the second equation: \begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l) \end{eqnarray}

He states the following:

This equation appears complicated, but each element has a nice interpretation. Suppose we know the error: \begin{eqnarray} \delta^{l+1} \end{eqnarray} At the l+1th layer. When we apply the transpose weight matrix, \begin{eqnarray} (w^{l+1})^T \end{eqnarray} We can think intuitively of this as moving the error backward through the network, giving us some sort of measure of the error at the output of the l'th layer. We then take the Hadamard product: \begin{eqnarray} \odot \sigma'(z^l) \end{eqnarray} This moves the error backward through the activation function in layer l, giving us the error δl in the weighted input to layer l.

I don't understand how taking the transpose of a weight-matrix moves the error backwards; Since we multiply by the weight matrix to get to the next layer, some sort of division by the weight matrix would be more logical for me to get from the l+1 layer to the l'th layer. I also do not really get why taking a hadamard product takes us further back, to the weighted input. Can somebody explain me what is going on here?

  • 1
    $\begingroup$ Think about it in terms of the chain rule. It is quite long to scribble this, but try to write not the update equation, but first the transformation equation (suppose the layer $i$ outputs something for layer $i+1$). Write the full set of equations in matrix form. Then differentiate with respect to the layer $i+1$ (maybe even pick it as the final layer) output without any regard for matrix dimensions. You should see how this all comes down. In the process - fit the dimensions of matrices so that derivatives match. That's it. $\endgroup$ Commented Jun 28, 2019 at 17:58

1 Answer 1


So lets do this computation carefully in for $\delta_K^{\ell-1}$ and hopefully you'll be able to see how to build the matrix out of it:

First, we use the chain rule to write $\delta_K^{\ell-1}$ in terms of $\delta_j^{\ell}$:

$$ \delta_K^{\ell-1} = \frac{\partial C}{\partial z_k^{\ell -1}} = \sum_j \frac{\partial C}{\partial z_j^{\ell}} \frac{\partial z_j^{\ell}}{\partial z_K^{\ell-1}} = \sum_j \delta_j^{\ell}\frac{\partial z_j^{\ell}}{\partial z_K^{\ell-1}} \,. $$

Now we just need to compute the partial derivatives. Since

$$ z^{\ell}_j = \sum_{k} w_{jk}^\ell \sigma(z_k^{\ell - 1}) + b_j^\ell\,, $$ the partial derivative is $$ \frac{\partial z_j^{\ell}}{\partial z_K^{\ell-1}} = w_{jK}\sigma'(z_{K}^{\ell-1})\,. $$ Putting these pieces together we have $$ \delta_K^{\ell-1} = \sum_j \delta_j^{\ell}w_{jK}\sigma'(z_{K}^{\ell-1}) = (\delta^\ell \cdot w_{\bullet K}) \times \sigma'(z_{K}^{\ell-1})\,. $$ Here, $w_{\bullet K})$ is the $k$'th column of $w_{jk}$ as a vector. So you see that the weights appear not as an inverted quantity, but as multiples of the error. If we write this using matrix notation the transpose just takes account of the convention that $\delta^{\ell}$ is a column vector.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.