What is the role of feed forward layer in Transformer Neural Network architecture? What is the role and purpose of the fully connected layer after the attention layer in Transformer architecture?
 A: The feed-forward layer is weights that is trained during training and the exact same matrix is applied to each respective token position.
Since it is applied without any communcation with or inference by other token positions it is a highly parallelizable part of the model.
The role and purpose is to process the output from one attention layer in a way to better fit the input for the next attention layer.
A: Consider encoder part of transformer.
If there is no feed-forward layer, self-attention is simply performing re-averaging of value vectors.
In order to add more model function, i.e. element-wise non-linearity transformation of incoming vectors, to transformer, we add feed-forward layer to encoder part of transformer.
A: Here is my version,
as @avata has said self attention blocks are simply performing re-average of values. Imagine in bert you have 144 self attention block (12 in each layer). If there is no FFN all will act the same and similar.
Adding FFN make each of them behave like a separate small model that can be trained (get parameters). Then the whole process become like training a "stacked ensemble learning" where each model get different weight. This is not the best analogy; but the purpose of FFN is to parameterize self-attention modules.
Each of FFN has 3072  hidden dimension in the Bert-base; this means a lot of parameters for learning Bert corresponds to FFN block. Therefore, there has been effort to optimize these modules (either by replacing them or by reducing their size)
