Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights).
Skip connections form conceptually a 'bus' which flows right the way through the network, and in reverse, the gradients can flow backwards along it too.
Each 'block' of network layers, such as conv layers, poolings, etc, taps the values at a point along the bus, and then adds/subtracts values onto the bus. This means that the blocks do affect the gradients, and conversely, affect the forward output values too. However, there is a direct connection through the network.
Actually, resnets ('residual networks') are not entirely well understood yet. They clearly work empirically. Some papers show they are like an ensemble of shallower networks. There are various theories :) Which are not necessarily self-contradictory. But either way, an explanation of exactly why they work is outside the scope of a Cross Validated question, being an open research question :)
I made a diagram of how I see resnets in my head, in an earlier answer, at Gradient backpropagation through ResNet skip connections . Here is the diagram I made, reproduced:
I understood the main concept, but how are these residual connections usually implemented? They remind me of how an LSTM unit works.
So, imagine a network where at each layer you have two conv blocks, in parallel:
- the input goes into each block
- the outputs are summed
Now, replace one of those blocks with a direct connection. An identity block if you like, or no block at all. That's a residual/skip connection.
In practice, the remaining conv unit would probably be two units in series, with an activation layer in between.
