# Why use both $\sin$ and $\cos$ functions in Transformer positional encoding?

I am interested why the positional encoding of Transformer use both $$\sin$$ and $$\cos$$, I understand use only $$\sin$$ will make all dimension equals to 0 in position 0.

But $$\cos$$ don't have such issue, there is no such position that all dimension are 0

Could anyone tell me the consideration of such design?

• What is the context here? What "Transformer" are you talking about? Commented Oct 23, 2019 at 2:57
• @JakeWestfall Transformers are a certain type of neural network model for sequential data. It's a common enough terminology in the neural networks literature; for example, pytorch includes it as a part of the standard library. pytorch.org/docs/stable/nn.html#transformer-layers
– Sycorax
Commented Oct 23, 2019 at 16:05

We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, $$PE_{pos+k}$$ can be represented as a linear function of $$PE_{pos}$$.
Indeed, $$\sin(x+k) = u\sin(x) + v \cos(x)$$ for some constants $$u, v$$, and likewise for $$\cos(x+k)$$, so this is true. If you only had $$\cos$$, it doesn't appear to me that you have this property.
• In general, linear functions are easy to learn. If, to attend to an offset $k$ from some position, I needed to compute some trigonometric or polynomial function of the inputs, then you might doubt the ability of the network to learn such a function. But linear just makes it trivially easy, since it can be done with one layer. Commented Jan 6, 2020 at 15:44