transformer, linear layer after attention My question regards this image:

It seems that after the multi head attention there is a linear layer as they mention also from here:

the linearity is given by the weights W^{o}. my quesion is: for the decoder, doesn't this linear layer mess up with the masking of the attention? I mean, if each token from the attention layer should not depend from the next tokens then these weights seem to mess up the whole thing or they don't? Indeed these linear weights will learn the dependencies among all the tokens and during inference could there be a problem, maybe(?). Thanks for any clarification.
 A: 
my quesion is: for the decoder, doesn't this linear layer mess up with the masking of the attention?

You know already that the answer is NO, because masked self attention goes before multi head attention with three inputs.

Indeed these linear weights will learn the dependencies among all the tokens and during inference could there be a problem, maybe(?).

I found that this paper you mentioned works with language translation task:

*

*Transformer simplified is Encoder (E) and Decoder (D) part.

*Both encoder and decoder do have FFNN (read: memory) and self attention (read: contextual memory)

*The model from the paper knows 50k words and can process 768 tokens from once (iteration)

*Training time words are encoded and go to both E and D.

*FFNN and Self Attention layers are learning while training.

*Inference time we put the foreign language sentence  (768 tokens max) to the E part and D part should start with empty sequence just having the start token and we should get the next word at red O, which we will use to improve our D token in the next step.


N was 6 in the paper so these are dense (linear) layers are learning the sentence language probabilities. Self attention layers are used to for instance to understand the pronouns. For instance in the sentence:
Chicken is walking on a road, it is ...

The "it" part should create attention to the Chicken and to the road.
Self attention is achieved using the dot product. There is another way mentioned in the paper, but it is not that efficient. The idea of the dot product is we actually have the word representation (vector space). This is where the values $V$ live.
If you multiply $QK^T$ this is how you enter that vector space and use softmax to select specific vector from this space that you will dot product with $V$.
To me masking acts as another regularizer that can be parametrized similar to label smoothing and residual dropout.
$\operatorname{Attention}(Q, K, V, M)=\operatorname{softmax}\left(\frac{Q K^{T}}{\sqrt{d_{k}}} M\right) V$
A: As far as I can understand, you are saying final Fully Connected Layer will consider the tokens of future words as well. Please see the image below:

As we can see in the right image, scaled dot product attention happens before the multiplication by W^o. Now see the left image. Inside the scaled dot product attention, masking is used to nullify the effect of the future positions. Thus, the future tokens are already nullified before final output fully connected layer.
A: The linear layer processes each element in the sequence independently, there is no interaction between them. Therefore information about future tokens is not available, also during training. Interaction/information exchange between tokens only occurs in the attention layers (hence we need masking)*
(* I have not really looked into the normalization layers but I guess that there is no interaction between elements there either. Note that transformers use layer normalization and not batch normalization where elements influence each other.)
