# What are “residual connections” in RNNs?

In Google's paper Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, it is stated

Our LSTM RNNs have $8$ layers, with residual connections between layers ...

What are residual connections? Why residual connections between layers?

Ideally, I am looking for a simple and intuitive explanation first, possibly accompanied by schematic representations.

The details can, of course, be found in the original papers, but I thought this question(s) would be beneficial to the community.

## 3 Answers

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights).

Skip connections form conceptually a 'bus' which flows right the way through the network, and in reverse, the gradients can flow backwards along it too.

Each 'block' of network layers, such as conv layers, poolings, etc, taps the values at a point along the bus, and then adds/subtracts values onto the bus. This means that the blocks do affect the gradients, and conversely, affect the forward output values too. However, there is a direct connection through the network.

Actually, resnets ('residual networks') are not entirely well understood yet. They clearly work empirically. Some papers show they are like an ensemble of shallower networks. There are various theories :) Which are not necessarily self-contradictory. But either way, an explanation of exactly why they work is outside the scope of a Cross Validated question, being an open research question :)

I made a diagram of how I see resnets in my head, in an earlier answer, at Gradient backpropagation through ResNet skip connections . Here is the diagram I made, reproduced:

I understood the main concept, but how are these residual connections usually implemented? They remind me of how an LSTM unit works.

So, imagine a network where at each layer you have two conv blocks, in parallel: - the input goes into each block - the outputs are summed

Now, replace one of those blocks with a direct connection. An identity block if you like, or no block at all. That's a residual/skip connection.

In practice, the remaining conv unit would probably be two units in series, with an activation layer in between.

• Edited answer to explain how the residual connections are implemented. They're simply 'identity'/'nop' connections. A network is a DAG (at least, in the absence of recurrent connections etc), and a resnet is no exception. – Hugh Perkins Jan 2 '18 at 2:10

For better and deeper understanding of the Residual Connection concept, you may want to also read this paper: Deep Residual Learning for Image Recognition. This is the same paper that is also referenced by "Attention Is All You Need" paper when explaining encoder element in the Transformers architecture.

in super-resolution there are many network architectures with residual connections. If you have a low-resolution picture x and you want to reconstruct a high resolution picture y, a network has to learn to not only predict the missing pixels from y, it also has to learn the representation of x.

Because x and y have a high correlation -> y is a higher resolution representation of x, you can add a skip connection from your input to the output of your last layer. That would mean, all the stuff happening in the network will only focus of learning y-x. Because at the end, x is added to the output.