What is the role and purpose of the fully connected layer after the attention layer in Transformer architecture?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Pretty amazing the best explanation for existence of feed-forward layers are vague statements, like "introduce nonlinearity" and "learn more complex patterns". Especially considering they account for roughly half of all model parameters $\endgroup$– DeLorean88Commented Jul 24, 2023 at 12:34
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
4
The feed-forward layer is weights that is trained during training and the exact same matrix is applied to each respective token position.
Since it is applied without any communcation with or inference by other token positions it is a highly parallelizable part of the model.
The role and purpose is to process the output from one attention layer in a way to better fit the input for the next attention layer.
-
2$\begingroup$ Thanks for the answer. Would you please elaborate further on your last sentence? Why and how would it make the attention output a better fit for the next layer? $\endgroup$– al palCommented Sep 6, 2020 at 21:12
-
2$\begingroup$ @alpal The simple answer is that you can't know for sure, I guess it's unique for respective model how the weights in the feed forward layer trains so the actual purpose isn't generic. The attention-logic is very dynamic but simple weight "postprocessing" adjustments is difficult for the model to learn and build into the attention logic. I am having difficulties finding the correct wordings for describing the necessity of the layer, it's an overall different logic if compared to attention and the combination of both is very dynamic. $\endgroup$ Commented Sep 7, 2020 at 9:14
-
6$\begingroup$ @alpal In multi-headed attention one purpose of the feed-forward layer is to process the concatenated output from the different attention heads. $\endgroup$ Commented Sep 7, 2020 at 9:26
-
$\begingroup$ I think the concatenated output from the multi-head Attention layer can be considered as analogy to flatten pixels of CNN. In general, we need the Neural Network to capture the complex interaction between Input and Output. $\endgroup$ Commented May 11 at 20:50
$\begingroup$
$\endgroup$
1
Consider encoder part of transformer.
If there is no feed-forward layer, self-attention is simply performing re-averaging of value vectors.
In order to add more model function, i.e. element-wise non-linearity transformation of incoming vectors, to transformer, we add feed-forward layer to encoder part of transformer.
-
$\begingroup$ yep, but still yet there are the q,k,v FFN projections that change the neuron's values. $\endgroup$ Commented Oct 26, 2023 at 14:39
$\begingroup$
$\endgroup$
Here is my version, as @avata has said self attention blocks are simply performing re-average of values. Imagine in bert you have 144 self attention block (12 in each layer). If there is no FFN all will act the same and similar.
Adding FFN make each of them behave like a separate small model that can be trained (get parameters). Then the whole process become like training a "stacked ensemble learning" where each model get different weight. This is not the best analogy; but the purpose of FFN is to parameterize self-attention modules.
Each of FFN has 3072 hidden dimension in the Bert-base; this means a lot of parameters for learning Bert corresponds to FFN block. Therefore, there has been effort to optimize these modules (either by replacing them or by reducing their size)
