This post about the Transformer introduced the concept of "Positional Encoding", while at the same time, the BERT paper mentioned "Position Embedding" as an input to BERT (e.g. in Figure 2).
First, are they different? Do they actually co-exist? If so (I guess so), what kind of difference do they have and why is it significant?
The original "Attention is all you need" paper use sine positional encoding. You can find a great in-depth explanation on this topic by Jonathan Kernes here: https://towardsdatascience.com/master-positional-encoding-part-i-63c05d90a0c3. While positional embedding is basically a learned positional encoding. Hope that it helps!
The positional encoding is a static function that maps an integer inputs to real-valued vectors in a way that captures the inherent relationships among the positions. That is, it captures the fact that position 4 in an input is more closely related to position 5 than it is to position 17.
While for the position embedding there will be plenty of training examples for the initial positions in our inputs and correspondingly fewer at the outer length limits. These latter embeddings may be poorly trained and may not generalize well during testing.
Reference: Speech and Language Processing
A word embedding is a learned look up map i.e. every word is given a one hot encoding which then functions as an index, and the corresponding to this index is a n dimensional vector where the coefficients are learn when training the model.
A positional embedding is similar to a word embedding. Except it is the position in the sentence is used as the index, rather than the one hot encoding.
A positional encoding is not learned but a chosen mathematical function. $\mathbb{N}\rightarrow\mathbb{R}^n$.
