Return to Answer

where the green numbers above the lines are indicating the forward pass, and the red numbers the backward pass(with the initial gradient 1).

where the blue numbers above the lines are indicating the forward pass, and the red numbers below the lines the backward pass(with the initial gradient 1).

where the green numbers above the lines indicate the forward pass, and the red numbers the backward pass(with the initial gradient 1).

where the blue numbers above the lines indicate the forward pass, and the red numbers below the lines the backward pass(with the initial gradient 1).

Here is the code for the last example:

# use tensorflow 1.12
x = tf.Variable(3, name='x', dtype=tf.float32)                                                      
y = tf.Variable(-4, name='y', dtype=tf.float32)                                                     
z = tf.Variable(2, name='z', dtype=tf.float32)                                                      
w = tf.Variable(-1, name='w', dtype=tf.float32)                                                     
                                                                                  
x_multiply_y = tf.math.multiply(x, y, name="x_multiply_y")                        
z_max_w = tf.math.maximum(z, w, name="z_max_w")                                   
xy_plus_zw = tf.math.add(z_max_w, x_multiply_y, name="xy_plus_zw") 
residual_op = tf.math.add(x_multiply_y, xy_plus_zw, name="residual_op")
multiply_2 = tf.math.multiply(residual_op, 2, name="multiply_2")  
# to make sure that the last gradient is 1 we make the cost 1                 
cost = multiply_2 + 45

optimizer = tf.train.AdamOptimizer()                                              
variables = tf.trainable_variables() 
all_ops = variables + [x_multiply_y, z_max_w, xy_plus_zw, residual_op, multiply_2]
gradients = optimizer.compute_gradients(cost, all_ops)
init = tf.global_variables_initializer() 
sess = tf.Session()
sess.run(init)
variables = [g[1] for g in gradients]
gradients = [g[0] for g in gradients]
gradients = sess.run(gradients)
for var, gdt in zip(variables, gradients):
    print(var.name, "\t", gdt)
# the results are here:
# x:0    -16.0
# y:0    12.0
# z:0    2.0
# w:0    0.0
# x_multiply_y:0     4.0
# z_max_w:0      2.0
# xy_plus_zw:0   2.0
# residual_op:0      2.0
# multiply_2:0   1.0

I'd like to recommend this limpid article: CS231n Convolutional Neural Networks for Visual Recognition, and let me compare the (simplified) vanilla network with the (simplified) residual network as follows.

Here is a diagram I borrowed from that page:

where the green numbers above the lines are indicating the forward pass, and the red numbers the backward pass(with the initial gradient 1).

And let's made a little change by adding a residual somewhere to get this one:

where the blue numbers above the lines are indicating the forward pass, and the red numbers bellow the lines the backward pass(with the initial gradient 1).

We can see that the gradients are accumulated from different sources.

HTH.

I'd like to recommend this limpid article: CS231n Convolutional Neural Networks for Visual Recognition, and let me compare the (simplified) vanilla network with the (simplified) residual network as follows.

Here is a diagram I borrowed from that page:

where the green numbers above the lines are indicating the forward pass, and the red numbers the backward pass(with the initial gradient 1).

And let's make a little change by adding a residual somewhere to get this one:

where the blue numbers above the lines are indicating the forward pass, and the red numbers below the lines the backward pass(with the initial gradient 1).

We can see that the gradients are accumulated from different sources.

HTH.

I recommend you read this article: CS231n Convolutional Neural Networks for Visual Recognition, and let me compare the (simplified) vanilla network with the (simplified) residual network as follows.

Here is a diagram I borrowed from that page:

where the green numbers above the lines are indicating the forward pass, and the red numbers the backward pass(with the initial gradient 1).

And let's made a little change by adding a residual somewhere to get this one:

where the blue numbers above the lines are indicating the forward pass, and the red numbers bellow the lines the backward pass(with the initial gradient 1).

We can see that the gradients are accumulated from different sources.

HTH.

I'd like to recommend this limpid article: CS231n Convolutional Neural Networks for Visual Recognition, and let me compare the (simplified) vanilla network with the (simplified) residual network as follows.

Here is a diagram I borrowed from that page:

where the green numbers above the lines are indicating the forward pass, and the red numbers the backward pass(with the initial gradient 1).

And let's made a little change by adding a residual somewhere to get this one:

where the blue numbers above the lines are indicating the forward pass, and the red numbers bellow the lines the backward pass(with the initial gradient 1).

We can see that the gradients are accumulated from different sources.

HTH.