# What is the intuition behind the positional cosine encoding in the transformer network?

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence.

What is the intuition behind it? It seems a bit counter intuitive to me to just add these values to the word embedding, they are 2 very different things.

Is the reasoning that it does not make sense for a single example but adding the same values over and over for thousands/millions of input sentences will enable the network to dissociate it?

Essentially the same word at different positions in the sentence will have slightly different embedding and this is where the network is able to capture the position information? It seems to me that it would be more intuitive to concatenate the cosine embedding rather than adding it.

Thanks a lot

• what do you mean by concatenation? May 22, 2019 at 17:28
• As in for a word embedding of dimension d, you would stack its positional encoding such that it becomes of dimension 2d, instead of using the addition operator.
– Tom
May 22, 2019 at 18:33

This procedure alters the angle between two embedding vectors. Suppose your word is embedded with a vector: $$e_1,e_2,\dots ,e_d$$. If there was no positional encoding then the angle between the embedding vectors of the same word will be always 0 regardless of the position of the word in a sentence.
Now, you alter the vector with positional encodings $$p_1,p_2,\dots ,p_d$$ and $$p'_1,p'_2,\dots ,p'_d$$ for two different positions in a sentence of the same word. Now the angle becomes: $$\cos(\alpha)=\frac{\sum_{i=1}^d(e_i+p_i)(e_i+p'_i)}{\sqrt{\left(\sum_{j=1}^d(e_j+p_j)^2\right)\left(\sum_{j=1}^d(e_j+p_j)^2\right)}}$$