# Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer?

Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar to with the output of the encoder? Especially because the original paper doesn't seem to mention it.

In particular (https://pytorch.org/tutorials/beginner/translation_transformer.html):

src = self.encoder(src) * math.sqrt(self.ninp)

# helper Module to convert tensor of input indices into corresponding tensor of token embeddings
class TokenEmbedding(nn.Module):
def __init__(self, vocab_size: int, emb_size):
super(TokenEmbedding, self).__init__()
self.embedding = nn.Embedding(vocab_size, emb_size)
self.emb_size = emb_size

def forward(self, tokens: Tensor):
return self.embedding(tokens.long()) * math.sqrt(self.emb_size)


Note that I am aware that the Attention layer has this equation:

$$\alpha = Attention(Q,K,V) = SoftMax( \frac{ Q K^\top }{\sqrt{D}} ) V$$

and they argue why about it in the paper in a one of the margins (something about sum of variance being 1).

Is this related to that comment and how is it related? Is this mentioned in the original paper?

cross posted:

• Good question. How come you didn't ask in /r/ML instead of these tier-2 subreddits?
– MWB
Sep 21, 2021 at 18:45
• @bobcat I think there are issues posting things there in weekdays. Since I don't work on weekends I never end up being able to post there. Sep 21, 2021 at 19:01

We multiply because we use learned embeddings to convert the input tokens and output tokens to vectors of dimension d_model in the embedding layer. Refer Section 3.4 on Embeddings and Softmax of the original paper. You'd also see that an object of TokenEmbedding is used in Seq2SeqTransformer for source and target token embeddings.
Note that the weights in word embeddings get initialized with zero mean and unit variance. Also, the embeddings get multiplied by sqrt(d) before being added to the positional encoding. The positional encoding is also on the same scale as the embeddings.