Deep NNs, backpropagation and error calculation [duplicate]

Question

This question already has an answer here:

• Deep NNs, backpropagation and error calculation 1 answer

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both in theory and practice with an actual implementation in python.

Now, something is not clear when he explains how to calculate the error on a specific layer in terms of error in the next layer (I will provide a screenshot because I don't want to mess with notation):

Here, $\delta^l$ is the error in layer l, $\odot$ is the elementwise product betweeen vectors, $w$ is a weight matrix, and $\sigma'$ indicates the derivative of an activation function.

Now, observe he points out that by applying the transpose of $w$ to the next-layer-error vector (which is the previous layer during backprop), one is "sending back" the error across the network, and then, multiplying by the derivative of sigma, one applies (I presume) the chain rule which completes the backprop step.

I have troubles understanding what does he really mean by saying that the traspose "sends back" the error. Why should it?
Perhaps he wants to apply the inverse trasformation? But unless the matrix happens to be orthogonal, which is not the case AFAIK, the transpose does NOT equal the inverse..
What is the algebraic significance of the transpose operator in this specific context?

In any case, I would have thought, reasoning about the chain rule, of applying some kind of differential operator to matrix $w$, instead of transposing it.

Before asking that question, I researched the topic for a long time, but I was unable to find whatever information about that particular point.

Opinions?

Answer 1

Short answer: the transpose is needed because you're looking at the inputs of all neurons in the next layer accepting outputs of neurons in this layer to calculate an error as a derivative. The original (untransposed) weight matrix expresses how outputs from previous layer are combined to make an input to the neurons in this layer.

Example: Say, you have two fully connected two neuron layers. Hence,

1. the weights that are involved when calculating the input into the first neuron of the second layer are: $w_{11},w_{21}$,
2. while the weights with which the output of the first neuron of the first layer gets into the inputs of neurons of the second layer are: $w_{11},w_{12}$. This is why transpose comes up.

One more thing. Why are we interested in #2? Because we're using them in the full derivative of $C$. We're looking at $\partial C/\partial a^l$, but we want to link it to the same thing in the next layer, so that the back propagation would be invoked. When we look at the derivative with respect to the output of a particular neuron $k$ in this layer, we use the full derivative trick: we express C as a function of inputs of all the neurons $i$ in the next layer that are accepting the output of our neuron $k$ as an input. When we do this the full derivative will be a sum: $$\frac{\partial C}{\partial a^l_k}=\sum_i\frac{\partial C}{\partial z^{l+1}_i}\frac{\partial z^{l+1}_i}{\partial a_k^l}$$ where obviously $z^{l+1}_i/\partial a_k^l=w^{l+1}_{ki}$ - the transposed weights.