How should one understand the keys, queries, and values that are often mentioned in attention mechanisms?

I've tried searching online, but all the resources I find only speak of them as if the reader already knows what they are.

Judging by the paper written by Bahdanau (Neural Machine Translation by Jointly Learning to Align and Translate), it seems as though values are the annotation vector $h$ but it's not clear as to what is meant by "query" and "key."

The paper that I mentioned states that attention is calculated by

$$c_i = \sum^{T_x}_{j = 1} \alpha_{ij} h_j$$


$$\alpha_{ij} = \frac{e^{e_{ij}}}{\sum^{T_x}_{k = 1} e^{ik}}$$

$$e_{ij} = a(s_{i - 1}, h_j)$$

Where are people getting the key, query, and value from these equations?

Thank you.

  • $\begingroup$ If this is the paper that you are talking about, it does not mention any "key", "query", or "value" for attention, and it seem to explain the symbols from the equations you quote, so I don't seem to understand what exactly is your question about? $\endgroup$ – Tim Aug 30 '19 at 12:45

I came from your other question Self-attention original work? The key/value/query formulation of attention is from the paper Attention Is All You Need.

How should one understand the queries, keys, and values

The key/value/query concepts come from retrieval systems. For example, when you type a query to search for some video on Youtube, the search engine will map your query against a set of keys (video title, description etc.) associated with candidate videos in the database, then present you the best matched videos (values).

The attention operation turns out can be thought of as a retrieval process as well, so the key/value/query concepts also apply here. (BTW the above example is just a toy system for illustration, in practice search engines and recommendation systems are much more complex.)

As mentioned in the paper you referenced (Neural Machine Translation by Jointly Learning to Align and Translate), attention by definition is just a weighted average of values,

$$c=\sum_{j}\alpha_jh_j$$ where $\sum \alpha_j=1$.

If we restrict $\alpha$ to be an one-hot vector, this operation becomes the same as retrieving from a set of elements $h$ with index $\alpha$. With the restriction removed, the attention operation can be thought of as doing "proportional retrieval" according to the probability vector $\alpha$.

It should be clear that $h$ in this context is the value. The difference between the two papers lies in how the probability vector $\alpha$ is calculated. The first paper (Bahdanau et al. 2015) computes the score through a neural network $$e_{ij}=a(s_i,h_j), \qquad a_{i,j}=\frac{\exp(e_{ij})}{\sum_k\exp(e_{ik})}$$ where $h_j$ is from the encoder sequence, and $s_i$ is from the decoder sequence. One problem of this approach is, say the encoder sequence is of length $m$ and the decoding sequence is of length $n$, we have to go through the network $m*n$ times to aqcuire all the attention scores $e_{ij}$.

A more efficient model would be to first project $s$ and $h$ onto a common space, then choose a similarity measure (e.g. dot product) as the attention score, like $$e_{ij}=f(s_i)g(h_j)^T$$ so we only have to compute $g(h_j)$ $m$ times and $f(s_i)$ $n$ times to get the projection vectors and $e_{ij}$ can be computed efficiently by matrix multiplication.

This is essentially the approach proposed by the second paper (Vaswani et al. 2017), where the two projection vectors are called query (for decoder) and key (for encoder), which is well aligned with the concepts in retrieval systems.

How are the queries, keys, and values obtained

The proposed multihead attention alone doesn't say much about how the queries, keys, and values are obtained as long as the dimension requirements are satisfied. They can come from different sources depending on the application scenario.

enter image description here

For unsupervised language model training like GPT, $Q, K, V$ are usually from the same source, so such operation is also called self-attention.

For the machine translation task in the second paper, it first applies self-attention separately to source and target sequences, then on top of that it applies another attention where $Q$ is from the target sequence and $K, V$ are from the source sequence.

For recommendation systems, $Q$ can be from the target items, $K, V$ can be from the user profile and history.

  • 1
    $\begingroup$ Hello. Thanks for the answer. Unfortunately, my question is how those values themselves are obtained (i.e. the Q, K, and V). I've read other blog posts (e.g. The Illustrated Transformer) and it's still unclear to me how the values are obtained from the context of the paper. For example, is Q simply the matrix product of the input X and some other weights? If so, then how are those weights obtained? $\endgroup$ – Seankala Aug 30 '19 at 1:45
  • $\begingroup$ Also, this question itself isn't actually pertaining to the calculation of Q, K, and V. Rather, I'm confused as to why the authors used different terminology compared to the original attention paper. $\endgroup$ – Seankala Aug 30 '19 at 1:45
  • $\begingroup$ @Seankala hi I made some updates for your questions, hope that helps $\endgroup$ – dontloo Aug 30 '19 at 6:48
  • $\begingroup$ Thanks a lot for this explanation! I still struggle to interprate the notation e_ij = a(s_i,h_j). So the neural network is a function of h_j and s_i, which are input sequences from the decoder and encoder sequences respectively. But what does the neural network look like? E.g. What are the target variables and what is the format of the input? $\endgroup$ – Emil Jan 17 at 13:43
  • $\begingroup$ @Emil hi, it is a sub-network of the whole, there are no specific target for it in training, it's usually trained jointly with the whole network wrt to the given task, in this case machine translation, more details in the A.1.2 ALIGNMENT MODEL section of the paper. $\endgroup$ – dontloo Jan 17 at 18:51

Where are people getting the key, query, and value from these equations?

The paper you refer to does not use such terminology as "key", "query", or "value", so it is not clear what you mean in here. There is no single definition of "attention" for neural networks, so my guess is that you confused two definitions from different papers.

In the paper, the attention module has weights $\alpha$ and the values to be weighted $h$, where the weights are derived from the recurrent neural network outputs, as described by the equations you quoted, and on the figure from the paper reproduced below.

![enter image description here

Similar thing happens in the Transformer model from the Attention is all you need paper by Vaswani et al, where they do use "keys", "querys", and "values" ($Q$, $K$, $V$). Vaswani et al define the attention cell differently:

$$ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\Big(\frac{QK^T}{\sqrt{d_k}}\Big)V $$

What they also use is multi-head attention, where instead of a single value for each $Q$, $K$, $V$, they provide multiple such values.

enter image description here

Where in the Transformer model, the $Q$, $K$, $V$ values can either come from the same inputs in the encoder (bottom part of the figure below), or from different sources in the decoder (upper right part of the figure). This part is crucial for using this model in translation tasks.

enter image description here

In both papers, as described, the values that come as input to the attention layers are calculated from the outputs of the preceding layers of the network. Both paper define different ways of obtaining those values, since they use different definition of attention layer.


See Attention is all you need - masterclass, from 15:46 onwards Lukasz Kaiser explains what q, K and V are.

So basically:

  • q = the vector representing a word
  • K and V = your memory, thus all the words that have been generated before. Note that K and V can be the same (but don't have to).

So what you do with attention is that you take your current query (word in most cases) and look in your memory for similar keys. To come up with a distribution of relevant words, the softmax function is then used.

New contributor
Emil is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.