# What are “residual connections” in RNNs?

In Google's paper Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, it is stated

Our LSTM RNNs have $8$ layers, with residual connections between layers ...

What are residual connections? Why residual connections between layers?

Ideally, I am looking for a simple and intuitive explanation first, possibly accompanied by schematic representations.

The details can, of course, be found in the original papers, but I thought this question(s) would be beneficial to the community.

• Possible duplicate of Gradient backpropagation through ResNet skip connections – Hugh Perkins Jan 1 '18 at 5:15
• @HughPerkins That question isn't exactly my question. The OP there doesn't even mention "residual connections", which is the main topic of this question. If "residual connections" and "skip connections" are the same thing, it isn't my fault. You either add it to your answer or, better, just give a better-phrased answer to my question explaining what residual connections are, given that your answer there is broken into pieces. By the way, you may want to rephrase your answer there so that to make it more sequential and fluid while reading. – nbro Jan 1 '18 at 13:19
• Yes, residual connections and skip connections are the same thing. I hadnt noticed they are actually different words actually, my brain simply interprets them as the same underlying 'thought vector' :) If you want to reassure yourself they are the same, you can see for example reddit.com/r/learnmachinelearning/comments/6b0uqs/… – Hugh Perkins Jan 1 '18 at 20:55
• I'm not sure why this is closed as unclear. I find it pretty clear. – amoeba Aug 22 '18 at 21:14

## 1 Answer

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights).

Skip connections form conceptually a 'bus' which flows right the way through the network, and in reverse, the gradients can flow backwards along it too.

Each 'block' of network layers, such as conv layers, poolings, etc, taps the values at a point along the bus, and then adds/subtracts values onto the bus. This means that the blocks do affect the gradients, and conversely, affect the forward output values too. However, there is a direct connection through the network.

Actually, resnets ('residual networks') are not entirely well understood yet. They clearly work empirically. Some papers show they are like an ensemble of shallower networks. There are various theories :) Which are not necessarily self-contradictory. But either way, an explanation of exactly why they work is outside the scope of a Cross Validated question, being an open research question :)

I made a diagram of how I see resnets in my head, in an earlier answer, at Gradient backpropagation through ResNet skip connections . Here is the diagram I made, reproduced:

I understood the main concept, but how are these residual connections usually implemented? They remind me of how an LSTM unit works.

So, imagine a network where at each layer you have two conv blocks, in parallel: - the input goes into each block - the outputs are summed

Now, replace one of those blocks with a direct connection. An identity block if you like, or no block at all. That's a residual/skip connection.

In practice, the remaining conv unit would probably be two units in series, with an activation layer in between.

• I understood the main concept, but how are these residual connections usually implemented? They remind me of how an LSTM unit works. – nbro Jan 1 '18 at 21:31
• Edited answer to explain how the residual connections are implemented. They're simply 'identity'/'nop' connections. A network is a DAG (at least, in the absence of recurrent connections etc), and a resnet is no exception. – Hugh Perkins Jan 2 '18 at 2:10