Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar to with the output of the encoder? Especially because the original paper doesn't seem to mention it.

In particular (https://pytorch.org/tutorials/beginner/translation_transformer.html):

src = self.encoder(src) * math.sqrt(self.ninp)

or this (https://pytorch.org/tutorials/beginner/transformer_tutorial.html):

# helper Module to convert tensor of input indices into corresponding tensor of token embeddings
class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size: int, emb_size):
        super(TokenEmbedding, self).__init__()
        self.embedding = nn.Embedding(vocab_size, emb_size)
        self.emb_size = emb_size

    def forward(self, tokens: Tensor):
        return self.embedding(tokens.long()) * math.sqrt(self.emb_size)

Note that I am aware that the Attention layer has this equation:

$$ \alpha = Attention(Q,K,V) = SoftMax( \frac{ Q K^\top }{\sqrt{D}} ) V $$

and they argue why about it in the paper in a one of the margins (something about sum of variance being 1).

Is this related to that comment and how is it related? Is this mentioned in the original paper?

cross posted:

  • $\begingroup$ Good question. How come you didn't ask in /r/ML instead of these tier-2 subreddits? $\endgroup$
    – MWB
    Sep 21, 2021 at 18:45
  • $\begingroup$ @bobcat I think there are issues posting things there in weekdays. Since I don't work on weekends I never end up being able to post there. $\endgroup$ Sep 21, 2021 at 19:01

2 Answers 2


We multiply because we use learned embeddings to convert the input tokens and output tokens to vectors of dimension d_model in the embedding layer. Refer Section 3.4 on Embeddings and Softmax of the original paper. You'd also see that an object of TokenEmbedding is used in Seq2SeqTransformer for source and target token embeddings.


Note that the weights in word embeddings get initialized with zero mean and unit variance. Also, the embeddings get multiplied by sqrt(d) before being added to the positional encoding. The positional encoding is also on the same scale as the embeddings.

My hypothesis is that the authors tried rescaling the embeddings by various numbers (as they certainly did with attention), and this particular rescaling happened to work, because it made the embeddings much bigger than the positional encodings (initially). The positional encodings are necessary, but they probably shouldn't be as "loud" as the words themselves.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.