I would like to ask about the last sentence here from paper https://arxiv.org/abs/1706.03762:

3.4 Embeddings and Softmax

Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{\text{model}}$. We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [30]. In the embedding layers, we multiply those weights by $\sqrt{d_{\text{model}}}$.

Why do we need to scale the embedding? (or do we even need? isn't it just some kind of normalization to only speed up the convergence? Because my own little experiments didn't really show any difference).

I didn't really find it in the whole paper and not even through exhaustive google search and it is not apparent to me. I know that there might not be anyone able to explain that so I am simultaneously sending mail to the authors and possibly answering my own question, because there might be other people wondering.


This is likely because the transformer shares the weights of the embedding layer and the output softmax. The scales you would use for the embeddings is different than the scale you use for a fully connected layer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.