4
$\begingroup$

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence.

What is the intuition behind it? It seems a bit counter intuitive to me to just add these values to the word embedding, they are 2 very different things.

Is the reasoning that it does not make sense for a single example but adding the same values over and over for thousands/millions of input sentences will enable the network to dissociate it?

Essentially the same word at different positions in the sentence will have slightly different embedding and this is where the network is able to capture the position information? It seems to me that it would be more intuitive to concatenate the cosine embedding rather than adding it.

Thanks a lot

$\endgroup$
  • $\begingroup$ what do you mean by concatenation? $\endgroup$ – Aksakal May 22 '19 at 17:28
  • $\begingroup$ As in for a word embedding of dimension d, you would stack its positional encoding such that it becomes of dimension 2d, instead of using the addition operator. $\endgroup$ – Tom May 22 '19 at 18:33
3
$\begingroup$

In positional encoding you encode the dimension with different frequency waves. Together with a position (on this wave) this gives you encoding that corresponds to each input. The encoding is subsequently added to the input.

This procedure alters the angle between two embedding vectors. Suppose your word is embedded with a vector: $e_1,e_2,\dots ,e_d$. If there was no positional encoding then the angle between the embedding vectors of the same word will be always 0 regardless of the position of the word in a sentence.

Now, you alter the vector with positional encodings $p_1,p_2,\dots ,p_d$ and $p'_1,p'_2,\dots ,p'_d$ for two different positions in a sentence of the same word. Now the angle becomes: $$\cos(\alpha)=\frac{\sum_{i=1}^d(e_i+p_i)(e_i+p'_i)}{\sqrt{\left(\sum_{j=1}^d(e_j+p_j)^2\right)\left(\sum_{j=1}^d(e_j+p_j)^2\right)}}$$

Depending on the difference in the position, the angle deviates more or less from zero.

Why not concatenate? Concatenation would not be merely change the angles. It would make the distances orthogonal dimensions. In the procedure above we're altering the vectors: perhaps, scaling their dimensions differently. This effectively alters their length and angles.

$\endgroup$
  • $\begingroup$ Thanks to the answer, I understand the process and what happens "technically" but I don't quite get the intuition behind it and why it makes sense. $\endgroup$ – Tom May 22 '19 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.