I would like to ask about the last sentence here from paper https://arxiv.org/abs/1706.03762:
3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{\text{model}}$. We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [30]. In the embedding layers, we multiply those weights by $\sqrt{d_{\text{model}}}$.
Why do we need to scale the embedding? (or do we even need? isn't it just some kind of normalization to only speed up the convergence? Because my own little experiments didn't really show any difference).
I didn't really find it in the whole paper and not even through exhaustive google search and it is not apparent to me. I know that there might not be anyone able to explain that so I am simultaneously sending mail to the authors and possibly answering my own question, because there might be other people wondering.
