I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer connections) but this one is asking specifically about back-propagation of gradients during training.

The basic architecture is here:

enter image description here

I read this paper, Study of Residual Networks for Image Recognition, and in Section 2 they talk about how one of the goals of ResNet is to allow a shorter/clearer path for the gradient to back-propagate to the base layer.

Can anyone explain how the gradient is flowing through this type of network? I don't quite understand how the addition operation, and lack of a parameterized layer after addition, allows for better gradient propagation. Does it have something to do with how the gradient doesn't change when flowing through an add operator and is somehow redistributed without multiplication?

Furthermore, I can understand how the vanishing gradient problem is alleviated if the gradient doesn't need to flow through the weight layers, but if theres no gradient flow through the weights then how do they get updated after the backward pass?

  • $\begingroup$ I didn't get your point the gradient doesn't need to flow through the weight layers, could you explain that? $\endgroup$
    – Anu
    Commented Feb 6, 2019 at 0:26

2 Answers 2


Add sends the gradient back equally to both inputs. You can convince yourself of this by running the following in tensorflow:

import tensorflow as tf

graph = tf.Graph()
with graph.as_default():
    x1_tf = tf.Variable(1.5, name='x1')
    x2_tf = tf.Variable(3.5, name='x2')
    out_tf = x1_tf + x2_tf

    grads_tf = tf.gradients(ys=[out_tf], xs=[x1_tf, x2_tf])
    with tf.Session() as sess:
        fd = {
            out_tf: 10.0
        print(sess.run(grads_tf, feed_dict=fd))


[1.0, 1.0]

So, the gradient will be:

  • passed back to previous layers, unchanged, via the skip-layer connection, and also
  • passed to the block with weights, and used to update those weights

Edit: there is a question: "what is the operation at the point where the highway connection and the neural net block join back together again, at the bottom of Figure 2?"

There answer is: they are summed. You can see this from Figure 2's formula:

$$ \mathbf{\text{output}} \leftarrow \mathcal{F}(\mathbf{x}) + \mathbf{x} $$

What this says is that:

  • the values in the bus ($\mathbf{x}$)
  • are added to the results of passing the bus values, $\mathbf{x}$, through the network, ie $\mathcal{F}(\mathbf{x})$
  • to give the output from the residual block, which I've labelled here as $\mathbf{\text{output}}$

Edit 2:

Rewriting in slightly different words:

  • in the forwards direction, the input data flows down the bus
    • at points along the bus, residual blocks can learn to add/remove values to the bus vector
  • in the backwards direction, the gradients flow back down the bus
    • along the way, the gradients update the residual blocks they move past
    • the residual blocks will themselves modify the gradients slightly too

The residual blocks do modify the gradients flowing backwards, but there are no 'squashing' or 'activation' functions that the gradients flow through. 'squashing'/'activation' functions are what causes the exploding/vanishing gradient problem, so by removing those from the bus itself, we mitigate this problem considerably.

Edit 3: Personally I imagine a resnet in my head as the following diagram. Its topologically identical to figure 2, but it shows more clearly perhaps how the bus just flows straight through the network, whilst the residual blocks just tap the values from it, and add/remove some small vector against the bus:

enter image description here

  • 1
    $\begingroup$ if the gradient is also being passed through the weight blocks (just like in regular networks) then where is the resnet benefit coming from? sure, it allows the gradient to skip directly to the base input but how does that offer a performance increase when the other path is still trained as normal? $\endgroup$
    – Simon
    Commented Mar 21, 2017 at 8:49
  • 5
    $\begingroup$ I see. So one gradient is skipping straight back to x, the other propagates through the weights back to x. do they get summed up when they reach x due to x having split into 2 paths? if so, isn't the gradient still changing as it moves back through these layers? $\endgroup$
    – Simon
    Commented Mar 21, 2017 at 9:00
  • 1
    $\begingroup$ The gradients flow all the way down the stack, unchanged. However, each block contributes its own gradient changes into the stack, after applying its weight updates, and generating its own set of gradients. Each block has both input and output, and gradients will flow out of the input, back into the gradient "highway". $\endgroup$ Commented Mar 21, 2017 at 9:03
  • 1
    $\begingroup$ @RonakAgrawal added an edit showing the sum operatoin from Figure 2, and explaining it $\endgroup$ Commented Dec 2, 2017 at 14:18
  • 1
    $\begingroup$ added a second edit rephrasing my explanation a bit :) $\endgroup$ Commented Dec 2, 2017 at 14:22

I'd like to recommend this limpid article: CS231n Convolutional Neural Networks for Visual Recognition, and let me compare the (simplified) vanilla network with the (simplified) residual network as follows.

Here is a diagram I borrowed from that page:

enter image description here

where the green numbers above the lines indicate the forward pass, and the red numbers the backward pass(with the initial gradient 1).

And let's make a little change by adding a residual somewhere to get this one:

enter image description here

where the blue numbers above the lines indicate the forward pass, and the red numbers below the lines the backward pass(with the initial gradient 1).

Here is the code for the last example:

# use tensorflow 1.12
x = tf.Variable(3, name='x', dtype=tf.float32)                                                      
y = tf.Variable(-4, name='y', dtype=tf.float32)                                                     
z = tf.Variable(2, name='z', dtype=tf.float32)                                                      
w = tf.Variable(-1, name='w', dtype=tf.float32)                                                     
x_multiply_y = tf.math.multiply(x, y, name="x_multiply_y")                        
z_max_w = tf.math.maximum(z, w, name="z_max_w")                                   
xy_plus_zw = tf.math.add(z_max_w, x_multiply_y, name="xy_plus_zw") 
residual_op = tf.math.add(x_multiply_y, xy_plus_zw, name="residual_op")
multiply_2 = tf.math.multiply(residual_op, 2, name="multiply_2")  
# to make sure that the last gradient is 1 we make the cost 1                 
cost = multiply_2 + 45

optimizer = tf.train.AdamOptimizer()                                              
variables = tf.trainable_variables() 
all_ops = variables + [x_multiply_y, z_max_w, xy_plus_zw, residual_op, multiply_2]
gradients = optimizer.compute_gradients(cost, all_ops)
init = tf.global_variables_initializer() 
sess = tf.Session()
variables = [g[1] for g in gradients]
gradients = [g[0] for g in gradients]
gradients = sess.run(gradients)
for var, gdt in zip(variables, gradients):
    print(var.name, "\t", gdt)
# the results are here:
# x:0    -16.0
# y:0    12.0
# z:0    2.0
# w:0    0.0
# x_multiply_y:0     4.0
# z_max_w:0      2.0
# xy_plus_zw:0   2.0
# residual_op:0      2.0
# multiply_2:0   1.0

We can see that the gradients are accumulated from different sources.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.