I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both in theory and practice with an actual implementation in python.
Now, something is not clear when he explains how to calculate the error on a specific layer in terms of error in the next layer (I will provide a screenshot because I don't want to mess with notation):
Here, $\delta^l$ is the error in layer l, $\odot$ is the elementwise product betweeen vectors, $w$ is a weight matrix, and $\sigma'$ indicates the derivative of an activation function.
Now, observe he points out that by applying the transpose of $w$ to the next-layer-error vector (which is the previous layer during backprop), one is "sending back" the error across the network, and then, multiplying by the derivative of sigma, one applies (I presume) the chain rule which completes the backprop step.
I have troubles understanding what does he really mean by saying that the traspose "sends back" the error. Why should it?
Perhaps he wants to apply the inverse trasformation? But unless the matrix happens to be orthogonal, which is not the case AFAIK, the transpose does NOT equal the inverse..
What is the algebraic significance of the transpose operator in this specific context?
In any case, I would have thought, reasoning about the chain rule, of applying some kind of differential operator to matrix $w$, instead of transposing it.
Before asking that question, I researched the topic for a long time, but I was unable to find whatever information about that particular point.
Opinions?