1
$\begingroup$

In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about an alternative approach. Can we simply concatenate a single integer to the word embeddings to represent positional information? Note that this way, we can keep the semantic and positional information separate. Normally, we combine these two by adding the word embeddings and positional embeddings together. Consequently, we need to strike a balance to ensure that the positional encoding does not overpower or dominate the word embedding information.

Please correct me if my understanding is wrong and let me about the issues with the proposed positional encoding by index idea.

$\endgroup$

2 Answers 2

3
$\begingroup$

Good question.

One problem with your proposed method of encoding is that, for lengthy sequences, your positional encoding integer can end up being quite large, out-scaling the other dimensions.

This article contains more reasons, and the most complete answer I've come across for this question as well as the chain of questions that ensues.

If we have a sequence of 500 tokens, we’ll end up with a 500 in our vector. In general, neural nets like their weights to hover around zero, and usually be equally balanced positive and negative. If not, you open yourself up to all sorts of problems, like exploding gradients and unstable training.

$\endgroup$
3
$\begingroup$

Great question. This has been shown to be possible, for example in Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2020)!

We use a simplified form of position embeddings where each “embedding” is simply a scalar that is added to the corresponding logit used for computing the attention weights.

Specifically, in their implementation, they use single scalar values added to the dot products in the attention layer (which can be interpreted as your idea of concatenating and then handling this dimension separately). Take a look at the HuggingFace implementation.

$\endgroup$
1
  • $\begingroup$ It's an interesting topic since single integer positional embedding idea is way simpler than the sinusoidal positional embeddings. Thanks for the readings :). However, I think there is more depth to this topic, I'll leave the question open for now. That's because single integer embeddings might be possible, but maybe sinusoidal ones are actually way more efficient/robust for some reason. $\endgroup$
    – Glue
    Commented Jul 12, 2023 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.