19
$\begingroup$

Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training.

This problem is circumvented by summing the input x to the result of a typical feed-forward computation in the following way:

$$ \mathcal F(x) + x = \left[ W_2 \sigma( W_1 \mathbf{x} + b_1 ) + b_2 \right] + \mathbf{x}.$$

This was schematically represented in [1] as:

enter image description here

On the other hand, it is also well known that transformer architectures have some residual networks, as the following picture elaborates:

enter image description here

Question: Residual connections are motivated in the context of very deep network architectures, but attention blocks perform very little computations compared to the networks that were outperformed in [1]; so, what is the motivation for the presence of shortcut connections in the attention-blocks of transformer architectures ?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ personally i throw resids on everything $\endgroup$
    – John Madden
    Commented Feb 21, 2022 at 7:08
  • 1
    $\begingroup$ @JohnMadden Even in shallow neural networks? $\endgroup$
    – Ramiro Hum-Sah
    Commented Feb 21, 2022 at 15:40
  • 2
    $\begingroup$ not reflexively in that case but if I run into trouble it's one of the things I would probably try (though in shallow networks I would probably try increasing the hidden layer widths to try to get asymptotic Gaussian Process behavior first). $\endgroup$
    – John Madden
    Commented Feb 21, 2022 at 17:11
  • $\begingroup$ That makes sense, thanks for your comment @JohnMadden $\endgroup$
    – Ramiro Hum-Sah
    Commented Feb 21, 2022 at 21:38
  • $\begingroup$ @RamiroHum-Sah: Can you please give me a link to the second of the illustrations above. I am especially interested in the annotations (which are not found in the original illustration from the Attention is all you need paper, right?) Thanks in advance. $\endgroup$
    – Hans-Peter Stricker
    Commented Nov 18, 2023 at 16:19

1 Answer 1

Reset to default
32
$\begingroup$

The reason for having the residual connection in Transformer is more technical than motivated by the architecture design.

Residual connections mainly help mitigate the vanishing gradient problem. During the back-propagation, the signal gets multiplied by the derivative of the activation function. In the case of ReLU, it means that in approximately half of the cases, the gradient is zero. Without the residual connections, a large part of the training signal would get lost during back-propagation. Residual connections reduce effect because summation is linear with respect to derivative, so each residual block also gets a signal that is not affected by the vanishing gradient. The summation operations of residual connections form a path in the computation graphs where the gradient does not get lost.

Another effect of residual connections is that the information stays local in the Transformer layer stack. The self-attention mechanism allows an arbitrary information flow in the network and thus arbitrary permuting the input tokens. The residual connections, however, always "remind" the representation of what the original state was. To some extent, the residual connections give a guarantee that contextual representations of the input tokens really represent the tokens.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Awesome, @Jindřich! Thanks for your very illuminating answer. I can't resist to ask about one detail in your last paragraph: Could you be so kind to elaborate a little bit more about what you mean by "The self-attention mechanism allows an arbitrary information flow in the network and thus arbitrary permuting the input tokens", please? What I understand is that perhaps the construction of (say) queries $Q=W_{Q}x$, for a given random matrix of weights $W_{Q}$ could give a very bad answer at the beginning of the training and the shortcut connection help us to improve that initial guesses ? $\endgroup$
    – Ramiro Hum-Sah
    Commented Feb 21, 2022 at 19:23
  • $\begingroup$ I think he meant that with residuals, self attention blocks can perform arbitrary computations without being concerned about preserving the original data. If residuals weren't used, the self attention block would need to ensure that it somehow preserved (at least some of) the original data. $\endgroup$
    – JMS
    Commented Aug 6, 2023 at 7:17
  • $\begingroup$ "training signal would get lost during back-propagation". Usually, residual blocks are added after many layers no just between two successive layers (as in the transformer architecture). So what is the contribution of skip connections to vanishing gradients in this case? $\endgroup$
    – Antonios Sarikas
    Commented Jun 26 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.