Look ahead bias in the skip connection of the Transformer-decoder/GPT2 architecture

Question

How come the residual connection on the attention module in Decoder-Transformers/GPT2 does not cause a look ahead bias?

This is my current understanding:

1. GPT is similar to the decoder side of the Transformer architecture. The masked self attention module and the Residual connection are the same as in the Transformer. However, the second self-attention is not present, and the input data is passed to it.
2. Attention on the decoder side uses a mask to avoid look ahead bias, i.e. input data attending any future data.
3. A Residual connection on an input x returns x + F(x) instead of F(x), where x is the input sequence. Residual connections are needed to stabilize back propagation. In this case, F(x) is masked attention, but x is not. This leads to my question:

Why doesn't the Residual connection cause a look ahead bias? Maybe because masking the attention is sufficient, for some reason?

Answer 1

The residual connection adds the sequence of input tokens to the sequence of output tokens. It works on a token by token base. So the resulting token at position n is simply the output token of the masked multihead attention at position n, plus the input token at position n (ONLY). Since the output token has no look ahead bias thanks to the masking and the input token has no look ahead bias, as it's only the unprocessed input token at position n, the residual connection doesn't introduce any look ahead bias. (This is also true for stacked attention layers as long as look ahead masks are used in each layer)