How does backpropagation in residual network works? i am studying the deep residual network currently, and i cannot fully understand how backpropagation in residual network works. Here some parts of the paper that i read.


so, how does the backpropagation works?
 A: Backpropagation is a very general algorithm can be applied anywhere where there is a computation graph on which you can define gradients. Residual networks, like simple fully connected networks, are computation graphs on which all the operations are differentiable and have mathematically defined gradients. Therefore, backprop works no differently there than on any other network.
Specifically, if each computation node $f_i$ takes inputs $y_i = (y_i^1, \ldots y_i^k)$ and computes $x_i = f_i(y_i)$ then all that is required for backprop are gradient functions $g_i^1, \ldots g_i^k$ such that $g_i^j(\frac{\partial L}{\partial x_i}, y_i)$ computes the portion of $\frac{\partial L}{\partial y_i^j}$ which "flows through" $f_i$. ($L$ is the loss)
Once that is done, you can reverse topological sort the graph, and then loop in order: for each node $f_i$, compute the result of each $g_i^j$ and add it to an accumulator variable stored in the corresponding input node of $f_i$ which will eventually contain the value $\frac{\partial L}{\partial y_i^j}$. Due to our topological sort, the gradient of that node will be correctly computed by the time we get to that node in our main loop. 
As an aside, i've always been puzzled by why virtually all introductions to backpropagation attempt to teach the topic by painfully stepping through the algorithm on a neural network and manually writing out all the chain rules, without ever stepping back to write down what the algorithm actually is! Once you have the general form, you realize there is no need to study "backpropagation through time" or "backpropagation through residual connections" or "backpropagation for batch norm" -- it's all just one algorithm.
