Why the positional embeddings for different positions are not confined

Question

for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))

we know PE for position i follows: [sin(i/denominator[0]),cos(i/denominator[0]), sin(i/denominator[1]),cos(i/denominator[1]), sin(i/denominator[2]),cos(i/denominator[2]),...]. note each element here is a proposed positional embedding element for an embedding element.

let's assume our embedding size is 1. so positional embedding for position i is sin(i/denominator[0]). also assume denominator[0] value is 0.2. so

PE(0,0)=sin(0)=0,
PE(1,0)=sin(5)=sin(1.59*Pi)=-.958,
PE(2,0)=sin(10)=sin(3.184*Pi)=-.544,
PE(3,0)=sin(15)=sin(4.777*Pi)=.6502
...

this just circulates the 2Pi. so in the sense of trying to assign a meaningful values for positional embeddings I dont realize how does it make sense. its not confined to sth like (i/max_sequence_length*pi/denominator[0]), that way if we used only cosine, the last words were assigned to -1 and first words assigned to 1. I mean some more understandable shifting embedding values into the space. if its not confined the positional embeddings dont obey a pattern the positional embeddings seem to be just random. how can not confining it makes sense?

Answer 1

That's the point of embeddings!

Using the time-series example from my answer to your other question, say that you want to create a weekly Fourier feature using the $\sin$ function as $\sin\big(\tfrac{2\pi t}{7}\big)$ where $t$ is the time index for days. The feature when plotted looks like below.

As you can see, I marked every 7th day with a red triangle and each of them has the same value of the "embedding" feature. That is the point, the feature represents 7th day with some numerical value. This enables us to use the feature in a regression model, where the numerical value would represent the day of the week.

The same in positional embeddings in the transformer model, the embeddings map the position of the word in the sentence to the embeddings space. Assuming that we are interested in "every seventh word" in a sentence (as in the time-series example above), we can use the embeddings to record the information. This enables you to model periodic trends in the data.

this just circulates the 2Pi. so in the sense of trying to assign a meaningful values for positional embeddings I dont realize how does it make sense. its not confined to sth like (i/max_sequence_length*pi/denominator[0]), that way if we used only cosine, the last words were assigned to -1 and first words assigned to 1.

If you did it like this, your model would be able to only work with texts of length smaller or equal to max_sequence_length, while we want it to work with texts of arbitrary length. Also, how would it differ from just using i as a feature? The point of positional embeddings is to encode cycles in the data.