# Why are residual connections needed in transformer architectures?

Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training.

This problem is circumvented by summing the input x to the result of a typical feed-forward computation in the following way:

$$\mathcal F(x) + x = \left[ W_2 \sigma( W_1 \mathbf{x} + b_1 ) + b_2 \right] + \mathbf{x}.$$

This was schematically represented in [1] as:

On the other hand, it is also well known that transformer architectures have some residual networks, as the following picture elaborates:

Question: Residual connections are motivated in the context of very deep network architectures, but attention blocks perform very little computations compared to the networks that were outperformed in [1]; so, what is the motivation for the presence of shortcut connections in the attention-blocks of transformer architectures ?

• personally i throw resids on everything Feb 21 at 7:08
• @JohnMadden Even in shallow neural networks? Feb 21 at 15:40
• not reflexively in that case but if I run into trouble it's one of the things I would probably try (though in shallow networks I would probably try increasing the hidden layer widths to try to get asymptotic Gaussian Process behavior first). Feb 21 at 17:11
• That makes sense, thanks for your comment @JohnMadden Feb 21 at 21:38

• Awesome, @Jindřich! Thanks for your very illuminating answer. I can't resist to ask about one detail in your last paragraph: Could you be so kind to elaborate a little bit more about what you mean by "The self-attention mechanism allows an arbitrary information flow in the network and thus arbitrary permuting the input tokens", please? What I understand is that perhaps the construction of (say) queries $Q=W_{Q}x$, for a given random matrix of weights $W_{Q}$ could give a very bad answer at the beginning of the training and the shortcut connection help us to improve that initial guesses ? Feb 21 at 19:23