Why Transformer use little activations?

Question

I am confused about the activation functions used in the famous NMT model Transformer. In other classical networks such as ResNet and LSTM, the activation functions are used after every linear transformation. So a stereotype image is that the activation functions should follow after the linear functions. But in the Transformer, I found that the activation functions are only used in Position-wise Feed-Forward Layers. In Multi-Head Attention Layers, there are only linear transformations. Can someone help to explain this? This blog (the annotated transformer) provides a reference for the implementation.

Answer 1

There are also softmax while calculating the attentions. But of course, the outputs become weights to the weighted sum of the value vectors, which is a linear calculation. But I think if you expand out these weights, here you get some non-linearity.