5
$\begingroup$

In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on).

This way, each cell contains semantic and positional information. However, since two types of information are represented by one number only, it seems unclear to which extent semantic information and positional information are represented in each cell.

For example:

word1 = [0.2, 0.1, -0.3]

Can be the sum of:

embedding1 = [-0.34,0.52,0.69]
positional_encoding1 = [0.54,-0.42,-0.99]

Or from:

embedding1 = [-0.64,-0.81,0.16]
positional_encoding1 = [0.84,0.91,0.14]

Hence, summing two cells makes no sense to me. Yet, Transformers are very effective algorithms, so I probably miss something. Is the problem that I describe a problem indeed? And can someone provide an intuitive explanation for summing the embedding and positional encoding?

$\endgroup$
1
  • 1
    $\begingroup$ Were you able to find an answer to this? I'm learning about transformers and immediately had this question as well. I wonder if maybe there's sufficient dimensionality in the embedding that the position encoding only has the effect of creating new variations of the original words rather than distorting the "meanings" of the words (i.e., shifting the embeddings in such a way to blur them with words with different meanings). Maybe any blurring effects are small relative to the effects of the increased dimensionality that would be required to account for position separately. $\endgroup$ Commented Mar 24, 2023 at 1:41

2 Answers 2

4
$\begingroup$

I've broken my head over this question for a long time.

Here's the best answer I came across, from redit.

TL;DR: It is intuitively possible that, in high dimensions, the word vectors form a smaller dimensional subspace within the full embedding space, and the positional vectors form a different smaller dimensional subspace approximately orthogonal to the one spanned by word vectors. Thus despite vector addition, the two subspaces can be manipulated essentially independently of each other by some single learned transformation.

If it is true that positional encodings are orthogonal to word embeddings, then that means that you can add a positional encoding vector to an embedding without "jumbling" the semantic information in the word embedding at all, and the model is able to discriminate positional information from semantic information just as well as if the positional information were concatenated to the embedding.

$\endgroup$
2
  • 2
    $\begingroup$ You might want to expand a little on your TL;DR, but it makes absolute sense (+1) ... and welcome to the site! $\endgroup$
    – jbowman
    Commented Apr 10 at 1:32
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Apr 10 at 2:36
3
$\begingroup$

Positional encoding is typically a deterministic function, so it's always the same on all inputs. You are adding the same vector to all of your inputs.

For example, if you were using a naive positional encoding, you would use

positional_encoding = [1, 2, 3, ..., n] for all inputs.

In Attention Is All You Need of course, they use a more sophisticated function, but it's still deterministic.

It can seem strange to add both embeddings together, but the Transformer isn't confused since the variability it sees in embedded inputs isn't caused by a blend of word and positional embeddings, it's only caused by the word embedding. Positional encoding changes every input in the same way.

$\endgroup$
4
  • $\begingroup$ I disagree with what you say. Although the positional encoding is deterministic, it still varies. Hence, the variability is caused by both the embedding and the positional encoding. $\endgroup$
    – Emil
    Commented Nov 7, 2022 at 8:49
  • $\begingroup$ Are you referring to a specific positional embedding method? It might help to know what you have in mind :) $\endgroup$
    – waxalas
    Commented Nov 7, 2022 at 15:42
  • $\begingroup$ No, the specific positional encoding is unrelated to my question. My point is that the resulting word embedding (word1), has two degrees of freedom. Hence, it is unknown how the positional encoding and original embedding influence this. $\endgroup$
    – Emil
    Commented Nov 9, 2022 at 11:02
  • $\begingroup$ Well, it's not unrelated. Try to compute it by hand and you'll see it doesn't vary. $\endgroup$
    – waxalas
    Commented Nov 9, 2022 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.