I have been trying to better understand backpropagation, so I decided to try to derive it for myself, but there is one step I'm not totally sure about. First explaining my notation
- $C_0$ is the loss function, e.g. $L_2$-loss.
- $a_j^{(L)}$ is the output of node $j$ in layer $(L)$, where layer $(L)$ is the output layer.
- $w_{jk}^{(L)}$ is the weight from node $k$ in layer $(L-1)$ to node $j$ in layer $(L)$.
- $g()^{(L)}$ is the activation function in layer $(L)$.
- $z_j^{(L)}$ is the weighted sum defined as $z_j^{(L)}= \sum_i w_{ji}^{(L)} a_i^{(L)}$, and $a_j^{(L)}=g(z_j^{(L)})$
I should also note that I'm only considering the effect of a single training sample.
So if we consider the weights between layer (L-m-1) and (L-m), where $m>0$, so we are somewhere in the middle of the network. Then the derivative with respect to a single weight in this layer is
$\frac{\partial C_0}{\partial w_{jk}^{(L-m)}} = \frac{\partial C_0}{\partial a_{j}^{(L-m)}}\frac{\partial a_j^{(L-m)}}{\partial z_{j}^{(L-m)}}\frac{\partial z_j^{(L-m)}}{\partial w_{jk}^{(L-m)}}$
This part, above, I'm confident in being correct, where the last two factors are straight forward to find. It is $\frac{\partial C_0}{\partial a_{j}^{(L-m)}}$ I'm not 100% sure how to find.
My attempt is
$ \frac{\partial C_0}{\partial a_{j}^{(L-m)}} = \sum_i \frac{\partial C_0}{\partial a_{i}^{(L-m+1)}}\frac{\partial a_i^{(L-m+1)}}{\partial z_{i}^{(L-m+1)}}\frac{\partial z_i^{(L-m+1)}}{\partial a_{j}^{(L-m)}} $
Where the $i$ runs over all the nodes in the layer (L-m+1). I got this using the chain rule for partial derivatives (here). Here everything is easy to compute or already computed for the layer above.
My question is, is my derivation correct? or did I mess up something in the partial derivatives? I'm not completely sure when it comes to the chain rule for partial derivatives.
