6
$\begingroup$

I am interested why the positional encoding of Transformer use both $\sin$ and $\cos$, I understand use only $\sin$ will make all dimension equals to 0 in position 0. enter image description here

But $\cos$ don't have such issue, there is no such position that all dimension are 0

enter image description here

Could anyone tell me the consideration of such design?

$\endgroup$
2
  • 1
    $\begingroup$ What is the context here? What "Transformer" are you talking about? $\endgroup$ Commented Oct 23, 2019 at 2:57
  • $\begingroup$ @JakeWestfall Transformers are a certain type of neural network model for sequential data. It's a common enough terminology in the neural networks literature; for example, pytorch includes it as a part of the standard library. pytorch.org/docs/stable/nn.html#transformer-layers $\endgroup$
    – Sycorax
    Commented Oct 23, 2019 at 16:05

1 Answer 1

11
$\begingroup$

The authors write

We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$.

Indeed, $\sin(x+k) = u\sin(x) + v \cos(x)$ for some constants $u, v$, and likewise for $\cos(x+k)$, so this is true. If you only had $\cos$, it doesn't appear to me that you have this property.

$\endgroup$
2
  • 2
    $\begingroup$ Maybe this is a newbie question, but, can you tell me how does linear transformation of positional encoding help the model to learn easily? I mean why is that property (linear transformation) useful ? $\endgroup$
    – Zephyr
    Commented Jan 6, 2020 at 15:35
  • 5
    $\begingroup$ In general, linear functions are easy to learn. If, to attend to an offset $k$ from some position, I needed to compute some trigonometric or polynomial function of the inputs, then you might doubt the ability of the network to learn such a function. But linear just makes it trivially easy, since it can be done with one layer. $\endgroup$
    – shimao
    Commented Jan 6, 2020 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.