Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding

Question

In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on).

This way, each cell contains semantic and positional information. However, since two types of information are represented by one number only, it seems unclear to which extent semantic information and positional information are represented in each cell.

For example:

word1 = [0.2, 0.1, -0.3]

Can be the sum of:

embedding1 = [-0.34,0.52,0.69]
positional_encoding1 = [0.54,-0.42,-0.99]

Or from:

embedding1 = [-0.64,-0.81,0.16]
positional_encoding1 = [0.84,0.91,0.14]

Hence, summing two cells makes no sense to me. Yet, Transformers are very effective algorithms, so I probably miss something. Is the problem that I describe a problem indeed? And can someone provide an intuitive explanation for summing the embedding and positional encoding?

Answer 1

I've broken my head over this question for a long time.

Here's the best answer I came across, from redit.

TL;DR: It is intuitively possible that, in high dimensions, the word vectors form a smaller dimensional subspace within the full embedding space, and the positional vectors form a different smaller dimensional subspace approximately orthogonal to the one spanned by word vectors. Thus despite vector addition, the two subspaces can be manipulated essentially independently of each other by some single learned transformation.

If it is true that positional encodings are orthogonal to word embeddings, then that means that you can add a positional encoding vector to an embedding without "jumbling" the semantic information in the word embedding at all, and the model is able to discriminate positional information from semantic information just as well as if the positional information were concatenated to the embedding.

Answer 2

Positional encoding is typically a deterministic function, so it's always the same on all inputs. You are adding the same vector to all of your inputs.

For example, if you were using a naive positional encoding, you would use

positional_encoding = [1, 2, 3, ..., n] for all inputs.

In Attention Is All You Need of course, they use a more sophisticated function, but it's still deterministic.

It can seem strange to add both embeddings together, but the Transformer isn't confused since the variability it sees in embedded inputs isn't caused by a blend of word and positional embeddings, it's only caused by the word embedding. Positional encoding changes every input in the same way.