Skip to main content
1 of 6
Lerner Zhang
  • 6.9k
  • 2
  • 44
  • 81

I recommend you read this article: CS231n Convolutional Neural Networks for Visual Recognition, and let me compare the vanilla network with the residual network as follows.

Here is a diagram I borrowed from that page:

enter image description here

where the green numbers above the lines are indicating the forward pass, and the red numbers the backward pass(with the initial gradient 1).

And let's made a little change by adding a residual somewhere to get this one:

enter image description here

where the blue numbers above the lines are indicating the forward pass, and the red numbers bellow the lines the backward pass(with the initial gradient 1).

We can see that the gradients are accumulated from different sources.

HTH.

Lerner Zhang
  • 6.9k
  • 2
  • 44
  • 81