I saw in some papers (like “Attention is all you need”) a block called: “Feed Forward Layer” or “Feed Forward Network”.

This is a simple block that contains FC -> Relu -> FC , and the main idea is the FC sizes, so for example: FC1 size is [256,2048] , FC2 size is: [2048,128], (may also be [2048,256] just an example).

So, instead of going to lower dimension straight ahead (means from 256 ->128), they first project to higher dimension (2048) and then they return to the “wanted size” (128).

Does anyone have an intuition about this block? I mean, I can understand why moving to higher dimension helps, but I need more than that, so if you have any intuition or tips to help better understand this block, It will be great.

Just for an example (the block is called here “Feed Forward”):

From "Attention is all you need", by Google.


Working with two layers and a non-linearity between them (FC1:[256,2048] , FC2: [2048,128]), enables you to have more complex features than just one FC: [256, 128], if only because you can have non-linear features.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.