Return to Question

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer connections) but this one is asking specifically about back-propagation of gradients during training.

The basic architecture is here:

I read this paper and in Section 2 they talk about how one of the goals of ResNet is to allow a shorter/clearer path for the gradient to back-propagate to the base layer.

Can anyone explain how the gradient is flowing through this type of network? I don't quite understand how the addition operation, and lack of a parameterized layer after addition, allows for better gradient propagation. Does it have something to do with how the gradient doesn't change when flowing through an add operator and is somehow redistributed without multiplication?

Furthermore, I can understand how the vanishing gradient problem is alleviated if the gradient doesn't need to flow through the weight layers, but if theres no gradient flow through the weights then how do they get updated after the backward pass?

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer connections) but this one is asking specifically about back-propagation of gradients during training.

The basic architecture is here:

I read this paper, Study of Residual Networks for Image Recognition, and in Section 2 they talk about how one of the goals of ResNet is to allow a shorter/clearer path for the gradient to back-propagate to the base layer.

Can anyone explain how the gradient is flowing through this type of network? I don't quite understand how the addition operation, and lack of a parameterized layer after addition, allows for better gradient propagation. Does it have something to do with how the gradient doesn't change when flowing through an add operator and is somehow redistributed without multiplication?

Furthermore, I can understand how the vanishing gradient problem is alleviated if the gradient doesn't need to flow through the weight layers, but if theres no gradient flow through the weights then how do they get updated after the backward pass?

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer connections) but this one is asking specifically about back-propagation of gradients during training.

The basic architecture is here:

I read this paper and in Section 2 they talk about how one of the goals of ResNet is to allow a shorter/clearer path for the gradient to back-propagate to the base layer.

Can anyone explain how the gradient is flowing through this type of network? I don't quite understand how the addition operation, and lack of a parameterized layer after addition, allows for better gradient propagation. Does it have something to do with how the gradient doesn't change when flowing through an add operator and is somehow redistributed without multiplication?

Furthermore, I can understand how the vanishing gradient problem is alleviated if the gradient doesn't need to flow through the weight layers, but if theres no gradient flow through the weights then how do they get updated after the backward pass?

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer connections) but this one is asking specifically about back-propagation of gradients during training.

The basic architecture is here:

I read this paper and in Section 2 they talk about how one of the goals of ResNet is to allow a shorter/clearer path for the gradient to back-propagate to the base layer.

Can anyone explain how the gradient is flowing through this type of network? I don't quite understand how the addition operation, and lack of a parameterized layer after addition, allows for better gradient propagation. Does it have something to do with how the gradient doesn't change when flowing through an add operator and is somehow redistributed without multiplication?

Furthermore, I can understand how the vanishing gradient problem is alleviated if the gradient doesn't need to flow through the weight layers, but if theres no gradient flow through the weights then how do they get updated after the backward pass?

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer connections) but this one is asking specifically about back-propagation of gradients during training.

The basic architecture is here:

I read this paper and in Section 2 they talk about how one of the goals of ResNet is to allow a shorter/clearer path for the gradient to back-propagate to the base layer.

Can anyone explain how the gradient is flowing through this type of network? I don't quite understand how the addition operation, and lack of a parameterized layer after addition, allows for better gradient propagation. Does it have something to do with how the gradient doesn't change when flowing through an add operator and is somehow redistributed without multiplication?

Furthermore, I can understand how the vanishing gradient problem is alleviated if the gradient doesn't need to flow through the weight layers, but if theres no gradient flow through the weights then how do they get updated after the backward pass?

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer connections) but this one is asking specifically about back-propagation of gradients during training.

The basic architecture is here:

I read this paper and in Section 2 they talk about how one of the goals of ResNet is to allow a shorter/clearer path for the gradient to back-propagate to the base layer.

Can anyone explain how the gradient is flowing through this type of network? I don't quite understand how the addition operation, and lack of a parameterized layer after addition, allows for better gradient propagation. Does it have something to do with how the gradient doesn't change when flowing through an add operator and is somehow redistributed without multiplication?

Furthermore, I can understand how the vanishing gradient problem is alleviated if the gradient doesn't need to flow through the weight layers, but if theres no gradient flow through the weights then how do they get updated after the backward pass?