# Why the positional embeddings for different positions are not confined

for this formulas PE(pos,2i)=sin(pos/(10000^(2i/modelDimension))) and PE(pos,2i+1)=cos(pos/(10000^(2i/modelDimension)))

we know PE for position i follows: [sin(i/denominator[0]),cos(i/denominator[0]), sin(i/denominator[1]),cos(i/denominator[1]), sin(i/denominator[2]),cos(i/denominator[2]),...]. note each element here is a proposed positional embedding element for an embedding element.

let's assume our embedding size is 1. so positional embedding for position i is sin(i/denominator[0]). also assume denominator[0] value is 0.2. so

PE(0,0)=sin(0)=0,
PE(1,0)=sin(5)=sin(1.59*Pi)=-.958,
PE(2,0)=sin(10)=sin(3.184*Pi)=-.544,
PE(3,0)=sin(15)=sin(4.777*Pi)=.6502
...


this just circulates the 2Pi. so in the sense of trying to assign a meaningful values for positional embeddings I dont realize how does it make sense. its not confined to sth like (i/max_sequence_length*pi/denominator[0]), that way if we used only cosine, the last words were assigned to -1 and first words assigned to 1. I mean some more understandable shifting embedding values into the space. if its not confined the positional embeddings dont obey a pattern the positional embeddings seem to be just random. how can not confining it makes sense?

That's the point of embeddings!

Using the time-series example from my answer to your other question, say that you want to create a weekly Fourier feature using the $$\sin$$ function as $$\sin\big(\tfrac{2\pi t}{7}\big)$$ where $$t$$ is the time index for days. The feature when plotted looks like below.

As you can see, I marked every 7th day with a red triangle and each of them has the same value of the "embedding" feature. That is the point, the feature represents 7th day with some numerical value. This enables us to use the feature in a regression model, where the numerical value would represent the day of the week.

The same in positional embeddings in the transformer model, the embeddings map the position of the word in the sentence to the embeddings space. Assuming that we are interested in "every seventh word" in a sentence (as in the time-series example above), we can use the embeddings to record the information. This enables you to model periodic trends in the data.

this just circulates the 2Pi. so in the sense of trying to assign a meaningful values for positional embeddings I dont realize how does it make sense. its not confined to sth like (i/max_sequence_length*pi/denominator[0]), that way if we used only cosine, the last words were assigned to -1 and first words assigned to 1.

If you did it like this, your model would be able to only work with texts of length smaller or equal to max_sequence_length, while we want it to work with texts of arbitrary length. Also, how would it differ from just using i as a feature? The point of positional embeddings is to encode cycles in the data.

• ur answer about the similarity of this type of embeddings make sense for this question and not the other one. because this fourier method apparently captures the seasonality (along sequence dimension relations) (I dont fully get it now but I suppose it does). I had guessed it right, to separate the question, because ur answer seemed to target this one. Commented Jun 12, 2023 at 12:52
• about while we want it to work with texts of arbitrary length, now we also pass the max_sequence_length to create embeddings. ofc I get that each row can be created independently this way (I mean rows of positional embeddings can be added to positional embeddings matrix this way) but in the way I suggest I cant. Commented Jun 12, 2023 at 12:55
• at least I know now I should study how fourier captures seasonality, to understand the answer to this question. thank u Commented Jun 12, 2023 at 12:56
• but one further question!! the fourier along the sequence dimension is for 1st data sin(2*pi*1), for 2nd cos(2*pi*2), for 3rd sin(2*pi*3) but here I have formula for i position is equal to sin(i/denominator[0]). note I mean the Alternating sin and cosine in fourier is along the sequence dimension but in my formula 1st thing: is Alternating in embedding dimension, 2nd thing: is not Alternating in sequence dimension. did I get the formula wrong or its the same as the original paper of attention is all u need? Commented Jun 12, 2023 at 13:03
• @FarhangAmaji you are not altering anything, it's sin((2 * pi * i) / m) where m is taken from a different value in denominator per each embedding (feature).
– Tim
Commented Jun 12, 2023 at 13:07