What is the intuition behind the positional cosine encoding in the transformer network?

Question

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence.

What is the intuition behind it? It seems a bit counter intuitive to me to just add these values to the word embedding, they are 2 very different things.

Is the reasoning that it does not make sense for a single example but adding the same values over and over for thousands/millions of input sentences will enable the network to dissociate it?

Essentially the same word at different positions in the sentence will have slightly different embedding and this is where the network is able to capture the position information? It seems to me that it would be more intuitive to concatenate the cosine embedding rather than adding it.

Thanks a lot

Answer 1

In positional encoding you encode the dimension with different frequency waves. Together with a position (on this wave) this gives you encoding that corresponds to each input. The encoding is subsequently added to the input.

This procedure alters the angle between two embedding vectors. Suppose your word is embedded with a vector: $e_1,e_2,\dots ,e_d$. If there was no positional encoding then the angle between the embedding vectors of the same word will be always 0 regardless of the position of the word in a sentence.

Now, you alter the vector with positional encodings $p_1,p_2,\dots ,p_d$ and $p'_1,p'_2,\dots ,p'_d$ for two different positions in a sentence of the same word. Now the angle becomes: $$\cos(\alpha)=\frac{\sum_{i=1}^d(e_i+p_i)(e_i+p'_i)}{\sqrt{\left(\sum_{j=1}^d(e_j+p_j)^2\right)\left(\sum_{j=1}^d(e_j+p_j)^2\right)}}$$

Depending on the difference in the position, the angle deviates more or less from zero.

Why not concatenate? Concatenation would not be merely change the angles. It would make the distances orthogonal dimensions. In the procedure above we're altering the vectors: perhaps, scaling their dimensions differently. This effectively alters their length and angles.