64
$\begingroup$

How should one understand the keys, queries, and values that are often mentioned in attention mechanisms?

I've tried searching online, but all the resources I find only speak of them as if the reader already knows what they are.

Judging by the paper written by Bahdanau (Neural Machine Translation by Jointly Learning to Align and Translate), it seems as though values are the annotation vector $h$ but it's not clear as to what is meant by "query" and "key."

The paper that I mentioned states that attention is calculated by

$$c_i = \sum^{T_x}_{j = 1} \alpha_{ij} h_j$$

with

$$ \begin{align} \alpha_{ij} & = \frac{e^{e_{ij}}}{\sum^{T_x}_{k = 1} e^{ik}} \\\\ e_{ij} & = a(s_{i - 1}, h_j) \end{align} $$

Where are people getting the key, query, and value from these equations?

Thank you.

$\endgroup$
  • $\begingroup$ If this is the paper that you are talking about, it does not mention any "key", "query", or "value" for attention, and it seem to explain the symbols from the equations you quote, so I don't seem to understand what exactly is your question about? $\endgroup$ – Tim Aug 30 '19 at 12:45
  • 3
    $\begingroup$ I was all confused by Q,K,V in attention, until I read this article: medium.com/@b.terryjack/…. It covers all questions from history to the recent implementation. I hope it will be useful for you too. $\endgroup$ – user185597 Mar 17 at 10:53
  • 2
    $\begingroup$ I am also looking into it. As far as I have understood, Query is also represented as "s" at some places. So it is output from the previous iteration of the decoder. And the key and value which are also represented as "h" at some places, is the word vector from the encoder. For reference, you can check youtube.com/watch?v=OyFJWRnt_AY and youtube.com/… Again I am still trying to understand more. Please do your further research and also lemme know if you find something. $\endgroup$ – Mohit Gahlot Apr 12 at 8:50
  • $\begingroup$ The best explanation for me: youtu.be/XXtpJxZBa2c?t=4337 $\endgroup$ – QtRoS Apr 18 at 10:45
  • $\begingroup$ @QtRoS I don't think it was explained there what the keys were, only what values and queries were. $\endgroup$ – Rainb Jun 25 at 6:24
62
+50
$\begingroup$

The key/value/query formulation of attention is from the paper Attention Is All You Need.

How should one understand the queries, keys, and values

The key/value/query concepts come from retrieval systems. For example, when you type a query to search for some video on Youtube, the search engine will map your query against a set of keys (video title, description etc.) associated with candidate videos in the database, then present you the best matched videos (values).

The attention operation turns out can be thought of as a retrieval process as well, so the key/value/query concepts also apply here. (BTW the above example is just a toy system for illustration, in practice search engines and recommendation systems are much more complex.)

As mentioned in the paper you referenced (Neural Machine Translation by Jointly Learning to Align and Translate), attention by definition is just a weighted average of values,

$$c=\sum_{j}\alpha_jh_j$$ where $\sum \alpha_j=1$.

If we restrict $\alpha$ to be an one-hot vector, this operation becomes the same as retrieving from a set of elements $h$ with index $\alpha$. With the restriction removed, the attention operation can be thought of as doing "proportional retrieval" according to the probability vector $\alpha$.

It should be clear that $h$ in this context is the value. The difference between the two papers lies in how the probability vector $\alpha$ is calculated. The first paper (Bahdanau et al. 2015) computes the score through a neural network $$e_{ij}=a(s_i,h_j), \qquad a_{i,j}=\frac{\exp(e_{ij})}{\sum_k\exp(e_{ik})}$$ where $h_j$ is from the encoder sequence, and $s_i$ is from the decoder sequence. One problem of this approach is, say the encoder sequence is of length $m$ and the decoding sequence is of length $n$, we have to go through the network $m*n$ times to aqcuire all the attention scores $e_{ij}$.

A more efficient model would be to first project $s$ and $h$ onto a common space, then choose a similarity measure (e.g. dot product) as the attention score, like $$e_{ij}=f(s_i)g(h_j)^T$$ so we only have to compute $g(h_j)$ $m$ times and $f(s_i)$ $n$ times to get the projection vectors and $e_{ij}$ can be computed efficiently by matrix multiplication.

This is essentially the approach proposed by the second paper (Vaswani et al. 2017), where the two projection vectors are called query (for decoder) and key (for encoder), which is well aligned with the concepts in retrieval systems. (There are later techniques to further reduce the computational complexity, for example Reformer, Linformer.)

How are the queries, keys, and values obtained

The proposed multihead attention alone doesn't say much about how the queries, keys, and values are obtained, they can come from different sources depending on the application scenario.

enter image description here

For unsupervised language model training like GPT, $Q, K, V$ are usually from the same source, so such operation is also called self-attention.

For the machine translation task in the second paper, it first applies self-attention separately to source and target sequences, then on top of that it applies another attention where $Q$ is from the target sequence and $K, V$ are from the source sequence.

For recommendation systems, $Q$ can be from the target items, $K, V$ can be from the user profile and history.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Hello. Thanks for the answer. Unfortunately, my question is how those values themselves are obtained (i.e. the Q, K, and V). I've read other blog posts (e.g. The Illustrated Transformer) and it's still unclear to me how the values are obtained from the context of the paper. For example, is Q simply the matrix product of the input X and some other weights? If so, then how are those weights obtained? $\endgroup$ – Seankala Aug 30 '19 at 1:45
  • $\begingroup$ Also, this question itself isn't actually pertaining to the calculation of Q, K, and V. Rather, I'm confused as to why the authors used different terminology compared to the original attention paper. $\endgroup$ – Seankala Aug 30 '19 at 1:45
  • 1
    $\begingroup$ @Seankala hi I made some updates for your questions, hope that helps $\endgroup$ – dontloo Aug 30 '19 at 6:48
  • $\begingroup$ Thanks a lot for this explanation! I still struggle to interprate the notation e_ij = a(s_i,h_j). So the neural network is a function of h_j and s_i, which are input sequences from the decoder and encoder sequences respectively. But what does the neural network look like? E.g. What are the target variables and what is the format of the input? $\endgroup$ – Emil Jan 17 at 13:43
  • 1
    $\begingroup$ @dontloo Thanks a lot! Best explanation ever! $\endgroup$ – alift Oct 23 at 0:46
13
$\begingroup$

I was also puzzled by the keys, queries, and values in the attention mechanisms for a while. After searching on the Web and digesting relevant information, I have a clear picture about how the keys, queries, and values work and why they would work!

Let's see how they work, followed by why they work.

In a seq2seq model, we encode the input sequence to a context vector, and then feed this context vector to the decoder to yield expected good output.

However, if the input sequence is long, relying on only one context vector become less effective. We need all the information from the hidden states in the input sequence (encoder) for better decoding (the attention mechanism).

One way to utilize the input hidden states is shown below: Image source: https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3 Image source: https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3

In other words, in this attention mechanism, the context vector is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key (this is a slightly modified sentence from [Attention Is All You Need] https://arxiv.org/pdf/1706.03762.pdf).

Here, the query is from the decoder hidden state, the key and value are from the encoder hidden states (key and value are the same in this figure). The score is the compatibility between the query and key, which can be a dot product between the query and key (or other form of compatibility). The scores then go through the softmax function to yield a set of weights whose sum equals 1. Each weight multiplies its corresponding values to yield the context vector which utilizes all the input hidden states.

Note that if we manually set the weight of the last input to 1 and all its precedences to 0s, we reduce the attention mechanism to the original seq2seq context vector mechanism. That is, there is no attention to the earlier input encoder states.

Now, let's consider the self-attention mechanism as shown in the figure below:

enter image description here Image source: https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a

The difference from the above figure is that the queries, keys, and values are transformations of the corresponding input state vectors. The others remain the same.

Note that we could still use the original encoder state vectors as the queries, keys, and values. So, why we need the transformation? The transformation is simply a matrix multiplication like this:

Query = I x W(Q)

Key = I x W(K)

Value = I x W(V)

where I is the input (encoder) state vector, and W(Q), W(K), and W(V) are the corresponding matrices to transform the I vector into the Query, Key, Value vectors.

What are the benefits of this matrix multiplication (vector transformation)?

The obvious reason is that if we do not transform the input vectors, the dot product for computing the weight for each input's value will always yield a maximum weight score for the individual input token itself. This may not be the desired case, say, for the pronoun token that we need it to attend to its referent.

Another less obvious but important reason is that the transformation may yield better representations for Query, Key, and Value. Recall the effect of Singular Value Decomposition (SVD) like that in the following figure:

Application of SVD

Image source: https://youtu.be/K38wVcdNuFc?t=10

By multiplying an input vector with a matrix V (from the SVD), we obtain a better representation for computing the compatibility between two vectors, if these two vectors are similar in the topic space as shown in the example in the figure.

And these matrices for transformation can be learned in a neural network!

In short, by multiplying the input vector with a matrix, we got:

  1. increase of the possibility for each input token to attend to other tokens in the input sequence, instead of individual token itself.

  2. possibly better (latent) representations of the input vector;

  3. conversion of the input vector into a space with a desired dimension, say, from dimension 5 to 2, or from n to m, etc (which is practically useful);

Note the transformation matrix is learnable (without manual setting).

I hope this help you understand the queries, keys, and values in the (self-)attention mechanism of deep neural networks.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The Python code to train the matrix look like this: ` x_Train = np.array([ [1, 1, 1, 0, 0], [3, 3, 3, 0, 0], ...[0, 1, 0, 2, 2] ]) y_Train = np.array([[1], [1], ..., [-1]]) x_Test = np.array([ [5, 0, 0, 0, 0], [0, 4, 5, 0, 0]]) y_Test = np.array([[1], [1]]) model = Sequential() model.add(Dense(units=2, input_dim=5, activation='tanh')) model.add(Dense(units=1,activation='tanh')) model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['mse']) model.fit(x=x_Train, y=y_Train, epochs=100) predictions = model.predict(x_Test)` $\endgroup$ – Sam Tseng Apr 28 at 18:34
  • 1
    $\begingroup$ this answer is very helpful! $\endgroup$ – Gregory Rut Sep 7 at 10:52
7
$\begingroup$

See Attention is all you need - masterclass, from 15:46 onwards Lukasz Kaiser explains what q, K and V are.

So basically:

  • q = the vector representing a word
  • K and V = your memory, thus all the words that have been generated before. Note that K and V can be the same (but don't have to).

So what you do with attention is that you take your current query (word in most cases) and look in your memory for similar keys. To come up with a distribution of relevant words, the softmax function is then used.

| cite | improve this answer | |
$\endgroup$
6
$\begingroup$

Tensorflow and Keras just expanded on their documentation for the Attention and AdditiveAttention layers. Here is a sneaky peek from the docs:

The meaning of query, value and key depend on the application. In the case of text similarity, for example, query is the sequence embeddings of the first piece of text and value is the sequence embeddings of the second piece of text. key is usually the same tensor as value.

But for my own explanation, different attention layers try to accomplish the same task with mapping a function $f: \Bbb{R}^{T\times D} \mapsto \Bbb{R}^{T \times D}$ where T is the hidden sequence length and D is the feature vector size. For the case of global self- attention which is the most common application, you first need sequence data in the shape of $B\times T \times D$, where $B$ is the batch size. Each forward propagation (particularly after an encoder such as a Bi-LSTM, GRU or LSTM layer with return_state and return_sequences=True for TF), it tries to map the selected hidden state (Query) to the most similar other hidden states (Keys). After repeating it for each hidden state, and softmax the results, multiply with the keys again (which are also the values) to get the vector that indicates how much attention you should give for each hidden state. I hope this helps anyone as it took me days to figure it out.

The flow of any attention layer

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

Where are people getting the key, query, and value from these equations?

The paper you refer to does not use such terminology as "key", "query", or "value", so it is not clear what you mean in here. There is no single definition of "attention" for neural networks, so my guess is that you confused two definitions from different papers.

In the paper, the attention module has weights $\alpha$ and the values to be weighted $h$, where the weights are derived from the recurrent neural network outputs, as described by the equations you quoted, and on the figure from the paper reproduced below.

![enter image description here

Similar thing happens in the Transformer model from the Attention is all you need paper by Vaswani et al, where they do use "keys", "querys", and "values" ($Q$, $K$, $V$). Vaswani et al define the attention cell differently:

$$ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\Big(\frac{QK^T}{\sqrt{d_k}}\Big)V $$

What they also use is multi-head attention, where instead of a single value for each $Q$, $K$, $V$, they provide multiple such values.

enter image description here

Where in the Transformer model, the $Q$, $K$, $V$ values can either come from the same inputs in the encoder (bottom part of the figure below), or from different sources in the decoder (upper right part of the figure). This part is crucial for using this model in translation tasks.

enter image description here

In both papers, as described, the values that come as input to the attention layers are calculated from the outputs of the preceding layers of the network. Both paper define different ways of obtaining those values, since they use different definition of attention layer.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.