1
$\begingroup$

I have been trying to better understand backpropagation, so I decided to try to derive it for myself, but there is one step I'm not totally sure about. First explaining my notation

  • $C_0$ is the loss function, e.g. $L_2$-loss.
  • $a_j^{(L)}$ is the output of node $j$ in layer $(L)$, where layer $(L)$ is the output layer.
  • $w_{jk}^{(L)}$ is the weight from node $k$ in layer $(L-1)$ to node $j$ in layer $(L)$.
  • $g()^{(L)}$ is the activation function in layer $(L)$.
  • $z_j^{(L)}$ is the weighted sum defined as $z_j^{(L)}= \sum_i w_{ji}^{(L)} a_i^{(L)}$, and $a_j^{(L)}=g(z_j^{(L)})$

I should also note that I'm only considering the effect of a single training sample.

So if we consider the weights between layer (L-m-1) and (L-m), where $m>0$, so we are somewhere in the middle of the network. Then the derivative with respect to a single weight in this layer is

$\frac{\partial C_0}{\partial w_{jk}^{(L-m)}} = \frac{\partial C_0}{\partial a_{j}^{(L-m)}}\frac{\partial a_j^{(L-m)}}{\partial z_{j}^{(L-m)}}\frac{\partial z_j^{(L-m)}}{\partial w_{jk}^{(L-m)}}$

This part, above, I'm confident in being correct, where the last two factors are straight forward to find. It is $\frac{\partial C_0}{\partial a_{j}^{(L-m)}}$ I'm not 100% sure how to find.

My attempt is

$ \frac{\partial C_0}{\partial a_{j}^{(L-m)}} = \sum_i \frac{\partial C_0}{\partial a_{i}^{(L-m+1)}}\frac{\partial a_i^{(L-m+1)}}{\partial z_{i}^{(L-m+1)}}\frac{\partial z_i^{(L-m+1)}}{\partial a_{j}^{(L-m)}} $

Where the $i$ runs over all the nodes in the layer (L-m+1). I got this using the chain rule for partial derivatives (here). Here everything is easy to compute or already computed for the layer above.

My question is, is my derivation correct? or did I mess up something in the partial derivatives? I'm not completely sure when it comes to the chain rule for partial derivatives.The type of network I'm considering

$\endgroup$
1
$\begingroup$

Your last equation seems correct and that is where the essence of back propagation algorithm lies in because you wrote error derivative with respect to some layer output in terms of the error derivative with respect to the next layer's output. This is a recurrence relation and enables efficient calculation of gradients. Otherwise, for each layer you'd have obtained several concentric summations all the way back to the output layer. Using this recursion, after calculating the gradients of the output layer, you can back-propagate these gradients up to the input layer.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.