How does a transformer network's attention mechanism determine what to pay attention to?

Question

The paper Attention is All You Need describes the transformer network which uses a "multi-head attention mechanism". The basic intuition behind this mechanism is that it weighs other tokens in a sequence by how much they are expected to influence the value of a token at some position. There are several high-level things I don't understand about it though and every article I read about them is either too technical or too hand-wavy for my experience level. These are the things I am trying to understand:

• Are the attention weights based solely on position of the tokens?
• Are the attention weights calculated using the vectors in the sequence? If so, which vectors are used?
• How is the attention mechanism trained? What are the inputs it received during training?

Answer 1

Are the attention weights based solely on position of the tokens?

No, they're based on the specific values that appear in the sequence. The attention formula actually has no information about position at all.

Because in natural language processing we believe the order of words is important, we will often incorporate a positional encoding into the vectors we provide to the attention function.

Are the attention weights calculated using the vectors in the sequence? If so, which vectors are used?

Yes, they are. A Transformer model contains several layers. At the lowest encoder layer, we use the source word embeddings (plus the positional encoding). At the next encoder layer, we use the sequence of outputs that this produced. This will be the same length as the input sequence.

The decoder layer attends to both (a) the vectors that the highest encoder layer produced and (b) the previous tokens in the decoder's generated sequence.

How is the attention mechanism trained? What are the inputs it received during training?

In terms of training, there's nothing special about the attention mechanism. It's just part of the model. The model is trained the same way you'd train any other feed-forward neural network. You provide the inputs (a source sentence), compute the outputs (a target sentence prediction), and compute the loss between the prediction and the true sequence. You then update the weights of the entire model according to the gradient of the loss.