# What are "residual connections" in RNNs?

In Google's paper Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, it is stated

Our LSTM RNNs have $8$ layers, with residual connections between layers ...

What are residual connections? Why residual connections between layers?

Ideally, I am looking for a simple and intuitive explanation first, possibly accompanied by schematic representations.

The details can, of course, be found in the original papers, but I thought this question(s) would be beneficial to the community.

• Possible duplicate of Gradient backpropagation through ResNet skip connections Jan 1, 2018 at 5:15
• Yes, residual connections and skip connections are the same thing. I hadnt noticed they are actually different words actually, my brain simply interprets them as the same underlying 'thought vector' :) If you want to reassure yourself they are the same, you can see for example reddit.com/r/learnmachinelearning/comments/6b0uqs/… Jan 1, 2018 at 20:55
• I'm not sure why this is closed as unclear. I find it pretty clear. Aug 22, 2018 at 21:14
• @HughPerkins residual connections are a subset of skip connections. Residual connections are basically $f(x) = x + g(x)$, while skip connection can use other forms of connection, e.g. concatenation Sep 23, 2021 at 19:52

## 4 Answers

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights).

Skip connections form conceptually a 'bus' which flows right the way through the network, and in reverse, the gradients can flow backwards along it too.

Each 'block' of network layers, such as conv layers, poolings, etc, taps the values at a point along the bus, and then adds/subtracts values onto the bus. This means that the blocks do affect the gradients, and conversely, affect the forward output values too. However, there is a direct connection through the network.

Actually, resnets ('residual networks') are not entirely well understood yet. They clearly work empirically. Some papers show they are like an ensemble of shallower networks. There are various theories :) Which are not necessarily self-contradictory. But either way, an explanation of exactly why they work is outside the scope of a Cross Validated question, being an open research question :)

I made a diagram of how I see resnets in my head, in an earlier answer, at Gradient backpropagation through ResNet skip connections . Here is the diagram I made, reproduced:

I understood the main concept, but how are these residual connections usually implemented? They remind me of how an LSTM unit works.

So, imagine a network where at each layer you have two conv blocks, in parallel: - the input goes into each block - the outputs are summed

Now, replace one of those blocks with a direct connection. An identity block if you like, or no block at all. That's a residual/skip connection.

In practice, the remaining conv unit would probably be two units in series, with an activation layer in between.

• Edited answer to explain how the residual connections are implemented. They're simply 'identity'/'nop' connections. A network is a DAG (at least, in the absence of recurrent connections etc), and a resnet is no exception. Jan 2, 2018 at 2:10
• Residual connections are a subset of skip connections. Residual connections are basically $f(x) = x + g(x)$, while skip connection can use other forms of connection, e.g. concatenation Sep 23, 2021 at 19:52

With respect to Deep Residual Learning for Image Recognition, I think it's correct to say that a ResNet contains both residual connections and skip connections, and that they are not the same thing.

Here's a quotation from the paper:

We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.

The concept of pushing the residual to zero indicates that the residual connection corresponds to layers that are learned rather than to the skip connection. I think it's best to understand a "ResNet" as a network that learns residuals.

In the following image (figure 2 from the paper), the path going through the weight layers and relu activation is the residual connection while the identity path is the skip connection.

The authors of Squeeze-and-Excitation Networks seem to have this understanding as well based on figure 3 from their paper.

References

For better and deeper understanding of the Residual Connection concept, you may want to also read this paper: Deep Residual Learning for Image Recognition. This is the same paper that is also referenced by "Attention Is All You Need" paper when explaining encoder element in the Transformers architecture.

in super-resolution there are many network architectures with residual connections. If you have a low-resolution picture x and you want to reconstruct a high resolution picture y, a network has to learn to not only predict the missing pixels from y, it also has to learn the representation of x.

Because x and y have a high correlation -> y is a higher resolution representation of x, you can add a skip connection from your input to the output of your last layer. That would mean, all the stuff happening in the network will only focus of learning y-x. Because at the end, x is added to the output.