288
$\begingroup$

How should one understand the keys, queries, and values that are often mentioned in attention mechanisms?

I've tried searching online, but all the resources I find only speak of them as if the reader already knows what they are.

Judging by the paper written by Bahdanau (Neural Machine Translation by Jointly Learning to Align and Translate), it seems as though values are the annotation vector $h$ but it's not clear as to what is meant by "query" and "key."

The paper that I mentioned states that attention is calculated by

$$c_i = \sum^{T_x}_{j = 1} \alpha_{ij} h_j$$

with

$$ \begin{align} \alpha_{ij} & = \frac{e^{e_{ij}}}{\sum^{T_x}_{k = 1} e^{ik}} \\\\ e_{ij} & = a(s_{i - 1}, h_j) \end{align} $$

Where are people getting the key, query, and value from these equations?

Thank you.

$\endgroup$
10
  • 3
    $\begingroup$ If this is the paper that you are talking about, it does not mention any "key", "query", or "value" for attention, and it seem to explain the symbols from the equations you quote, so I don't seem to understand what exactly is your question about? $\endgroup$
    – Tim
    Aug 30, 2019 at 12:45
  • 13
    $\begingroup$ I was all confused by Q,K,V in attention, until I read this article: medium.com/@b.terryjack/…. It covers all questions from history to the recent implementation. I hope it will be useful for you too. $\endgroup$
    – user185597
    Mar 17, 2020 at 10:53
  • 3
    $\begingroup$ I am also looking into it. As far as I have understood, Query is also represented as "s" at some places. So it is output from the previous iteration of the decoder. And the key and value which are also represented as "h" at some places, is the word vector from the encoder. For reference, you can check youtube.com/watch?v=OyFJWRnt_AY and youtube.com/… Again I am still trying to understand more. Please do your further research and also lemme know if you find something. $\endgroup$ Apr 12, 2020 at 8:50
  • 1
    $\begingroup$ The best explanation for me: youtu.be/XXtpJxZBa2c?t=4337 $\endgroup$
    – QtRoS
    Apr 18, 2020 at 10:45
  • $\begingroup$ @QtRoS I don't think it was explained there what the keys were, only what values and queries were. $\endgroup$
    – Rainb
    Jun 25, 2020 at 6:24

11 Answers 11

261
+50
$\begingroup$

The key/value/query formulation of attention is from the paper Attention Is All You Need.

How should one understand the queries, keys, and values

The key/value/query concept is analogous to retrieval systems. For example, when you search for videos on Youtube, the search engine will map your query (text in the search bar) against a set of keys (video title, description, etc.) associated with candidate videos in their database, then present you the best matched videos (values).

The attention operation can be thought of as a retrieval process as well.

As mentioned in the paper you referenced (Neural Machine Translation by Jointly Learning to Align and Translate), attention by definition is just a weighted average of values,

$$c=\sum_{j}\alpha_jh_j$$ where $\sum \alpha_j=1$.

If we restrict $\alpha$ to be a one-hot vector, this operation becomes the same as retrieving from a set of elements $h$ with index $\alpha$. With the restriction removed, the attention operation can be thought of as doing "proportional retrieval" according to the probability vector $\alpha$.

It should be clear that $h$ in this context is the value. The difference between the two papers lies in how the probability vector $\alpha$ is calculated. The first paper (Bahdanau et al. 2015) computes the score through a neural network $$e_{ij}=a(s_i,h_j), \qquad \alpha_{i,j}=\frac{\exp(e_{ij})}{\sum_k\exp(e_{ik})}$$ where $h_j$ is from the encoder sequence, and $s_i$ is from the decoder sequence. One problem of this approach is, say the encoder sequence is of length $m$ and the decoding sequence is of length $n$, we have to go through the network $m*n$ times to acquire all the attention scores $e_{ij}$.

A more efficient model would be to first project $s$ and $h$ onto a common space, then choose a similarity measure (e.g. dot product) as the attention score, like $$e_{ij}=f(s_i)g(h_j)^T$$ so we only have to compute $g(h_j)$ $m$ times and $f(s_i)$ $n$ times to get the projection vectors and $e_{ij}$ can be computed efficiently by matrix multiplication.

This is essentially the approach proposed by the second paper (Vaswani et al. 2017), where the two projection vectors are called query (for decoder) and key (for encoder), which is well aligned with the concepts in retrieval systems. (There are later techniques to further reduce the computational complexity, for example Reformer, Linformer, FlashAttention.)

How are the queries, keys, and values obtained

The proposed multihead attention alone doesn't say much about how the queries, keys, and values are obtained, they can come from different sources depending on the application scenario.

$$ \begin{align}\text{MultiHead($Q$, $K$, $V$)} & = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W^{O} \\ \text{where head$_i$} & = \text{Attention($QW_i^Q$, $KW_i^K$, $VW_i^V$)} \end{align}$$ Where the projections are parameter matrices: $$ \begin{align} W_i^Q & \in \mathbb{R}^{d_\text{model} \times d_k}, \\ W_i^K & \in \mathbb{R}^{d_\text{model} \times d_k}, \\ W_i^V & \in \mathbb{R}^{d_\text{model} \times d_v}, \\ W_i^O & \in \mathbb{R}^{hd_v \times d_{\text{model}}}. \end{align}$$

For unsupervised language model training like GPT, $Q, K, V$ are usually from the same source, so such operation is also called self-attention.

For the machine translation task in the second paper, it first applies self-attention separately to source and target sequences, then on top of that it applies another attention where $Q$ is from the target sequence and $K, V$ are from the source sequence.

For recommendation systems, $Q$ can be from the target items, $K, V$ can be from the user profile and history.

$\endgroup$
12
  • 18
    $\begingroup$ Hello. Thanks for the answer. Unfortunately, my question is how those values themselves are obtained (i.e. the Q, K, and V). I've read other blog posts (e.g. The Illustrated Transformer) and it's still unclear to me how the values are obtained from the context of the paper. For example, is Q simply the matrix product of the input X and some other weights? If so, then how are those weights obtained? $\endgroup$
    – Sean
    Aug 30, 2019 at 1:45
  • 2
    $\begingroup$ @Seankala hi I made some updates for your questions, hope that helps $\endgroup$
    – dontloo
    Aug 30, 2019 at 6:48
  • 2
    $\begingroup$ Thanks a lot for this explanation! I still struggle to interprate the notation e_ij = a(s_i,h_j). So the neural network is a function of h_j and s_i, which are input sequences from the decoder and encoder sequences respectively. But what does the neural network look like? E.g. What are the target variables and what is the format of the input? $\endgroup$
    – Emil
    Jan 17, 2020 at 13:43
  • 8
    $\begingroup$ "The key/value/query formulation of attention is from the paper Attention Is All You Need" <-- this is not correct and is confusing. This paper most definitely already assumes you know how the Q,K,V attention mechanism works, its contribution is that it ONLY uses that mechanism and not any LSTMs or recurrent networks as was previously used for translation. $\endgroup$
    – BjornW
    Sep 18, 2022 at 17:58
  • 2
    $\begingroup$ @BjornW oh I didn't know that, so where is the earliest definition of QKV attention from? $\endgroup$
    – dontloo
    Sep 19, 2022 at 15:26
74
$\begingroup$

I was also puzzled by the keys, queries, and values in the attention mechanisms for a while. After searching on the Web and digesting relevant information, I have a clear picture about how the keys, queries, and values work and why they would work!

Let's see how they work, followed by why they work.

Attention to replace context vector

In a seq2seq model, we encode the input sequence to a context vector, and then feed this context vector to the decoder to yield expected good output.

However, if the input sequence becomes long, relying on only one context vector become less effective. We need all the information from the hidden states in the input sequence (encoder) for better decoding (the attention mechanism).

One way to utilize the input hidden states is shown below: Image source: https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3 Image source

In other words, in this attention mechanism, the context vector is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key (this is a slightly modified sentence from Attention Is All You Need).

Here, the query is from the decoder hidden state, the key and value are from the encoder hidden states (key and value are the same in this figure). The score is the compatibility between the query and key, which can be a dot product between the query and key (or other form of compatibility). The scores then go through the softmax function to yield a set of weights whose sum equals 1. Each weight multiplies its corresponding values to yield the context vector which utilizes all the input hidden states.

Note that if we manually set the weight of the last input to 1 and all its precedences to 0s, we reduce the attention mechanism to the original seq2seq context vector mechanism. That is, there is no attention to the earlier input encoder states.

Self-Attention uses Q, K, V all from the input

Now, let's consider the self-attention mechanism as shown in the figure below:

enter image description here Image source

The difference from the above figure is that the queries, keys, and values are transformations of the corresponding input state vectors. The others remain the same.

Note that we could still use the original encoder state vectors as the queries, keys, and values. So, why we need the transformation? The transformation is simply a matrix multiplication like this:

Query = I x W(Q)

Key = I x W(K)

Value = I x W(V)

where I is the input (encoder) state vector, and W(Q), W(K), and W(V) are the corresponding matrices to transform the I vector into the Query, Key, Value vectors.

What are the benefits of this matrix multiplication (vector transformation)?

The obvious reason is that if we do not transform the input vectors, the dot product for computing the weight for each input's value will always yield a maximum weight score for the individual input token itself. In other words, when we compute the n attention weights (j for j=1, 2, ..., n) for input token at position i, the weight at i (j==i) is always the largest than the other weights at j=1, 2, ..., n (j<>i). This may not be the desired case. For example, for the pronoun token, we need it to attend to its referent, not the pronoun token itself.

Another less obvious but important reason is that the transformation may yield better representations for Query, Key, and Value. Recall the effect of Singular Value Decomposition (SVD) like that in the following figure:

Application of SVD

Image source

By multiplying an input vector with a matrix V (from the SVD), we obtain a better representation for computing the compatibility between two vectors, if these two vectors are similar in the topic space as shown in the example in the figure.

And these matrices for transformation can be learned in a neural network!

In short, by multiplying the input vector with a matrix, we got:

  1. increase of the possibility for each input token to attend to other tokens in the input sequence, instead of individual token itself

  2. possibly better (latent) representations of the input vector

  3. conversion of the input vector into a space with a desired dimension, say, from dimension 5 to 2, or from n to m, etc (which is practically useful)

I hope this help you understand the queries, keys, and values in the (self-)attention mechanism of deep neural networks.

$\endgroup$
4
  • 1
    $\begingroup$ @Sam Teens, thank you. So, could we use the same encoder hidden states (say, LSTM sequences) as inputs to calculate Q, K, and V? Is this the self part of the attention? $\endgroup$ Jul 9, 2021 at 11:41
  • $\begingroup$ Yes, of course. You can apply the self-attention mechanism in a seq2seq network based on LSTM. $\endgroup$
    – Sam Tseng
    Feb 26, 2022 at 13:34
  • $\begingroup$ I had trouble following the "Latent Semantic Indexing" image and tried to work out was meant in davidvandebunte.gitlab.io/executable-notes/notes/se/…. $\endgroup$ Aug 21, 2022 at 20:38
  • $\begingroup$ Wow - amazing way to explain the basis for attention while also connecting it to dimensionality reduction and LSI. Mind blown! $\endgroup$
    – Yu Chen
    Dec 2, 2022 at 1:55
62
$\begingroup$

Big picture

Basically Transformer builds a graph network where a node is a position-encoded token in a sequence.

During training:

  1. Get un-connected tokens as a sequence (e.g. sentence).
  2. Wires connections among tokens by having looked at the co-occurrences of them in billions of sequences.

What roles Q and K will play to build this graph network? You could be Q in your society trying to build the social graph network with other people. Each person in the people is K and you will build the connections with them. Eventually by having billions of interactions with other people, the connections become dependent on the contexts even with the same person K.

You may be superior to a person K at work, but K may be a master of martial art for you. As you remember such connections/relations with others based on the contexts, Transformer model (trained on a specific dataset) figures out such context dependent connections from Q to K (or from you to other person(s)), which is a memory that it offers.

If the layers go up higher, your individual identity as K will be blended into larger parts via going through the BoW process which plays the role.

With regard to the Markov Chain (MC), there is only one static connection from Q to K as P(K|Q) in MC as MC does not have the context memory that Transformer model offers.

First, understand Q and K

First, focus on the objective of First MatMul in the Scaled dot product attention using Q and K.

enter image description here

Intuition on what is Attention

For the sentence "jane visits africa".

When your eyes see jane, your brain looks for the most related word in the rest of the sentence to understand what jane is about (query). Your brain focuses or attends to the word visit (key).

This process happens for each word in the sentence as your eyes progress through the sentence.

First MatMul as Inquiry System using Vector Similarity

The first MatMul implements an inquiry system or question-answer system that imitates this brain function, using Vector Similarity Calculation. Watch CS480/680 Lecture 19: Attention and Transformer Networks by professor Pascal Poupart to understand further.

Think about the attention essentially being some form of approximation of SELECT that you would do in the database.
enter image description here

enter image description here

Think of the MatMul as an inquiry system that processes the inquiry: "For the word q that your eyes see in the given sentence, what is the most related word k in the sentence to understand what q is about?" The inquiry system provides the answer as the probability.

q k probability
jane visit 0.94
visit africa 0.86
africa visit 0.76

Note that the softmax is used to normalize values into probabilities so that their sum becomes 1.0.

enter image description here

There are multiple ways to calculate the similarity between vectors such as cosine similarity. Transformer attention uses simple dot product.

Where are Q and K from

The transformer encoder training builds the weight parameter matrices WQ and Wk in the way Q and K builds the Inquiry System that answers the inquiry "What is k for the word q".

The calculation goes like below where x is a sequence of position-encoded word embedding vectors that represents an input sentence.

  1. Picks up a word vector (position encoded) from the input sentence sequence, and transfer it to a vector space Q. This becomes the query.
    $Q = X \cdot W_{Q}^T$

  2. Pick all the words in the sentence and transfer them to the vector space K. They become keys and each of them is used as key.
    $K = X \cdot W_K^T$

  3. For each (q, k) pair, their relation strength is calculated using dot product.
    $q\_to\_k\_similarity\_scores = matmul(Q, K^T)$

  4. Weight matrices $W_Q$ and $W_K$ are trained via the back propagations during the Transformer training.

We first needs to understand this part that involves Q and K before moving to V.

enter image description here

Borrowing the code from Let's build GPT: from scratch, in code, spelled out. by Andrej Karpathy.

# B: Batch size
# T: Sequence length or max token size e.g. 512 for BERT. 'T' because of 'Time steps = Sequence length'
# D: Dimensions of the model embedding vector, which is d_model in the paper.
# H or h: Number of multi attention heads in Multi-head attention


def calculate_dot_product_similarities(
        query: Tensor,
        key: Tensor,
) -> Tensor:
    """
    Calculate similarity scores between queries and keys using dot product.

    Args:
        query: embedding vector of query of shape (B, h, T, d_k)
        key: embedding vector of key of shape (B, h, T, d_k)

    Returns: Similarities (closeness) between q and k of shape (B, h, T, T) where
        last (T, T) represents relations between all query elements in T sequence
        against all key elements in T sequence. If T is people in an organization,
        (T,T) represents all (cartesian product) social connections among them.
        The relation considers d_k number of features.
    """
    # --------------------------------------------------------------------------------
    # Relationship between k and q as the first MatMul using dot product similarity:
    # (B, h, T, d_k) @ (B, hH, d_k, T) ---> (B, h, T, T)
    # --------------------------------------------------------------------------------
    similarities = query @ key.transpose(-2, -1)            # dot product
    return similarities                                     # shape:(B, h, T, T)

Then, understand how V is created using Q and K

Second Matmul

Self Attention then generates the embedding vector called attention value as a bag of words (BoW) where each word contributes proportionally according to its relationship strength to q. This occurs for each q from the sentence sequence. The embedding vector is encoding the relations from q to all the words in the sentence.

Citing the words from Andrej Karpathy:

What is the easiest way for tokens to communicate. The easiest way is just average.

He makes it simple for the sake of tutorial but the essence is BoW.

enter image description here

def calculate_attention_values(
        similarities,
        values
):
    """
    For every q element, create a Bag of Words that encodes the relationships with
    other elements (including itself) in T, using (q,k) relationship value as the
    strength of the relationships.

    Citation:
    > On each of these projected versions of queries, keys and values we then perform
    > the attention function in parallel, yielding d_v-dimensional output values.

    ```
    bows = []
    for row in similarities:                    # similarity matrix of shape (T,T)
        bow = sum([                             # bow:shape(d_v,)
            # each column in row is (q,k) similarity score s
            s*v for (s,v) in zip(row,values)    # k:shape(), v:shape(d_v,)
=        ])
        bows.append(bow)                        # bows:shape(T,d_v)
    ```

    Args:
        similarities: q to k relationship strength matrix of shape (B, h, T, T)
        values: elements of sequence with length T of shape (B, h, T, d_v)

    Returns: Bag of Words for every q element of shape (B, h, T, d_v)
    """
    return similarities @ values     # (B,h,T,T) @ (B,h,T,d_v) -> (B,h,T,d_v)

References

There are multiple concepts that will help understand how the self attention in transformer works, e.g. embedding to group similars in a vector space, data retrieval to answer query Q using the neural network and vector similarity.

$\endgroup$
5
  • 3
    $\begingroup$ This is of course a silly question, but the dot product of "jane" with "jane" would always be 1, so why do you have 0.01 for jane * jane? $\endgroup$ Feb 21, 2022 at 19:40
  • 1
    $\begingroup$ @cheesus, because one 'jane' is from K and the other 'jane' is from Q so they are from different spaces. Hence the "Where are Q and K are from" part is there. $\endgroup$
    – mon
    Feb 28, 2022 at 10:51
  • 3
    $\begingroup$ Yeah ok, thank you this is very good for Qs and Ks, however you never justify why we can "forget about V". I still am very confused on what Vs are and why they are even considered. $\endgroup$
    – alelom
    Jul 15, 2022 at 10:30
  • $\begingroup$ Dear Mon, can you please tell me where this screenshot from the point Where are Q and K from from? (i.stack.imgur.com/DWNTr.jpg) is taken from? I try to understand the relationship between W_q, W_k and W_v. I can post a few quotes on them that describe how each of them underlies a specific training process such that their weights differ in a way which makes the queries, so says the source, for example more susceptible for capturing features or grammatical relationships such as prepositions and the tokens they attend to, while keys ... $\endgroup$
    – von spotz
    Jun 30, 2023 at 18:33
  • $\begingroup$ Well, take one of the quotes in full length. There are quite some helpful quotes on the projection matrices like this one: Queries provide selectivity and focus, allowing the model to attend to relevant information, while keys facilitate the establishment of connections and context, aiding in the understanding and generation of meaningful outputs. Together, they work in tandem to enable the attention mechanism to effectively process the input embeddings and perform various tasks. Also: keys are responsible for establishing connections and relationships between queries and input embeddings. $\endgroup$
    – von spotz
    Jun 30, 2023 at 18:37
18
$\begingroup$

I'm going to try provide an English text example. The following is based solely on my intuitive understanding of the paper 'Attention is all you need'.

Say you have a sentence:

  • I like Natural Language Processing , a lot !

Assume that we already have input word vectors for all the 9 tokens in the previous sentence. So, 9 input word vectors.

Looking at the encoder from the paper 'Attention is all you need', the encoder needs to produce 9 output vectors, one for each word. This is done, through the Scaled Dot-Product Attention mechanism, coupled with the Multi-Head Attention mechanism. I'm going to focus only on an intuitive understanding of the Scaled Dot-Product Attention mechanism, and I'm not going to go into the scaling mechanism.

Walking through an example for the first word 'I':

  • The query is the input word vector for the token "I"

  • The keys are the input word vectors for all the other tokens, and for the query token too, i.e (semi-colon delimited in the list below):

    [like;Natural;Language;Processing;,;a;lot;!] + [I]

  • The word vector of the query is then DotProduct-ed with the word vectors of each of the keys, to get 9 scalars / numbers a.k.a "weights"

  • These weights are then scaled, but this is not important to understand the intuition

  • The weights then go through a 'softmax' which is a particular way of normalizing the 9 weights to values between 0 and 1. This becomes important to get a "weighted-average" of the value vectors , which we see in the next step.

  • Finally, the initial 9 input word vectors a.k.a values are summed in a "weighted average", with the normalized weights of the previous step. This final step results in a single output word vector representation of the word "I"

Now that we have the process for the word "I", rinse and repeat to get word vectors for the remaining 8 tokens. We now have 9 output word vectors, each put through the Scaled Dot-Product attention mechanism. You can then add a new attention layer/mechanism to the encoder, by taking these 9 new outputs (a.k.a "hidden vectors"), and considering these as inputs to the new attention layer, which outputs 9 new word vectors of its own. And so on ad infinitum.

If this Scaled Dot-Product Attention layer summarizable, I would summarize it by pointing out that each token (query) is free to take as much information using the dot-product mechanism from the other words (values), and it can pay as much or as little attention to the other words as it likes by weighting the other words with (keys) . The real power of the attention layer / transformer comes from the fact that each token is looking at all the other tokens at the same time (unlike an RNN / LSTM which is restricted to looking at the tokens to the left)

The Multi-head Attention mechanism in my understanding is this same process happening independently in parallel a given number of times (i.e number of heads), and then the result of each parallel process is combined and processed later on using math. I didn't fully understand the rationale of having the same thing done multiple times in parallel before combining, but i wonder if its something to do with, as the authors might mention, the fact that each parallel process takes place in a separate Linear Algebraic 'space' so combining the results from multiple 'spaces' might be a good and robust thing (though the math to prove that is way beyond my understanding...)

$\endgroup$
4
  • 1
    $\begingroup$ From these two statements, quoting: 1) The keys are the input word vectors for all the other tokens, and for the query token too and 2) the initial 9 input word vectors a.k.a values we can only logically conclude that Keys and Values are the same thing. But then why two different names? Unfortunately, your explanation remains confusing. $\endgroup$
    – alelom
    Jul 15, 2022 at 10:12
  • 1
    $\begingroup$ They are indeed the same thing. They have two different names because they serve two different functions. The keys serve as weights for the attention mechanism. The values are what the context vector for the query is derived from...weighted by the keys. There is some 'self-attention' in there, basically, with each word in a sentence attending to all the other words in the sentence (and itself) $\endgroup$
    – Nitin
    Jul 15, 2022 at 16:58
  • $\begingroup$ In the code, q, k and v are all different vectors. So values and keys are not the same. github.com/karpathy/nanoGPT/blob/master/model.py#L64 $\endgroup$
    – yters
    May 24, 2023 at 4:41
  • $\begingroup$ Actually, even in the code they are. Look at line 43 in the github which produces the q,k,v. You'll see that the self.c_attn variable is actually a linear projection layer which is duplicating the vector 'x' 3 times, each with a size n_emb which i guess is the word embedding size of 'x' $\endgroup$
    – Nitin
    May 25, 2023 at 9:45
17
$\begingroup$

See Attention is all you need - masterclass, from 15:46 onwards Lukasz Kaiser explains what q, K and V are.

So basically:

  • q = the vector representing a word
  • K and V = your memory, thus all the words that have been generated before. Note that K and V can be the same (but don't have to).

So what you do with attention is that you take your current query (word in most cases) and look in your memory for similar keys. To come up with a distribution of relevant words, the softmax function is then used.

$\endgroup$
5
  • 9
    $\begingroup$ then why do we need both K and V? why not only K? $\endgroup$
    – xtiger
    Aug 10, 2021 at 12:52
  • 2
    $\begingroup$ i am with xtiger. This is not clear at all Quote from the paper "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key." That means K and V are DIFERRENT. $\endgroup$
    – Long
    Aug 23, 2021 at 14:44
  • 1
    $\begingroup$ @xtiger you could use V=K, but in the general lookup case, you usually do not. For example, if we had a recipe lookup for Q="pizza", we may retrieve the ingredients or the recipe for how to make a pizza. In both of these cases, V would have a dimension much larger than the Q (or K). Indeed, if you look at the specifications in the other postings above, you will see that Q and K have to be of the same dimension, but V can be of a different (often larger) dimension. $\endgroup$
    – kfmfe04
    Dec 30, 2021 at 19:48
  • $\begingroup$ I added an answer at the very last to this question a year ago, offering a situation where people are not using the same K and V. (sorry for the late update, I just got enough reputation to comment here...) $\endgroup$
    – Flora Sun
    Jul 8, 2022 at 4:18
  • $\begingroup$ @kfmfe04 Hey, I am thinking about your pizza case and I like the idea of it. Though in the end you mentioned that "V can be of a different dimension" and may I ask why this is possible using the dot-product attention? Since Q will be a weighted sum of V and weights are computed basing on dot-product. So shouldn't them be at least broadcastable? So how could V be in higher dimension? Projection? $\endgroup$
    – Flora Sun
    Aug 8, 2022 at 19:14
13
$\begingroup$

Tensorflow and Keras just expanded on their documentation for the Attention and AdditiveAttention layers. Here is a sneaky peek from the docs:

The meaning of query, value and key depend on the application. In the case of text similarity, for example, query is the sequence embeddings of the first piece of text and value is the sequence embeddings of the second piece of text. key is usually the same tensor as value.

But for my own explanation, different attention layers try to accomplish the same task with mapping a function $f: \Bbb{R}^{T\times D} \mapsto \Bbb{R}^{T \times D}$ where T is the hidden sequence length and D is the feature vector size. For the case of global self- attention which is the most common application, you first need sequence data in the shape of $B\times T \times D$, where $B$ is the batch size. Each forward propagation (particularly after an encoder such as a Bi-LSTM, GRU or LSTM layer with return_state and return_sequences=True for TF), it tries to map the selected hidden state (Query) to the most similar other hidden states (Keys). After repeating it for each hidden state, and softmax the results, multiply with the keys again (which are also the values) to get the vector that indicates how much attention you should give for each hidden state. I hope this helps anyone as it took me days to figure it out.

The flow of any attention layer

$\endgroup$
12
$\begingroup$
  • Queries is a set of vectors you want to calculate attention for.
  • Keys is a set of vectors you want to calculate attention against.
  • As a result of dot product multiplication you'll get set of weights a (also vectors) showing how attended each query against Keys. Then you multiply it by Values to get resulting set of vectors.

enter image description here

Now let's look at word processing from the article "Attention is all you need". There are two self-attending (xN times each) blocks, separately for inputs and outputs plus cross-attending block transmitting knowledge from inputs to outputs.

Each self-attending block gets just one set of vectors (embeddings added to positional values). In this case you are calculating attention for vectors against each other. So Q=K=V. You just need to calculate attention for each q in Q.

Cross-attending block transmits knowledge from inputs to outputs. In this case you get K=V from inputs and Q are received from outputs. I think it's pretty logical: you have database of knowledge you derive from the inputs and by asking Queries from the output you extract required knowledge.

How attention works: dot product between vectors gets bigger value when vectors are better aligned. Then you divide by some value (scale) to evade problem of small gradients and calculate softmax (when sum of weights=1). At this point you get set of weights sum=1 that tell you for which vectors in Keys your query is better aligned. All that's left is to multiply by Values.

$\endgroup$
3
  • $\begingroup$ Like in many other answers, Queries and Keys are clearly defined, whereas Values are not. What are Values? $\endgroup$
    – alelom
    Jul 15, 2022 at 10:15
  • $\begingroup$ After experimenting with self-attention, I think that q and K is kinda like when go to library and librarian instead of recommending you one specific book, provides you with a huge table how related your query to each book. "This book is about pirates, just like your query, is", says librarian, "but it's not about young pirates, just rather old and constantly nagging". You get this table of comparisons and use it to inspect the library. $\endgroup$ Jul 16, 2022 at 13:02
  • 1
    $\begingroup$ But there is one thing to keep in mind: this explanation is vague since whole Q-K-V idea is more explanatory than something from real life. You don't actually work with Q-K-V, you work with partial linear representations (nn.Linear within multi-head attention splits the data between heads). And data is totally different from initial vector representations after first block already, so you don't compare word against other words like in every explanation on the web, it's more like a universal computing unit used to efficiently extract knowledge. $\endgroup$ Jul 16, 2022 at 13:15
11
$\begingroup$

Where are people getting the key, query, and value from these equations?

The paper you refer to does not use such terminology as "key", "query", or "value", so it is not clear what you mean in here. There is no single definition of "attention" for neural networks, so my guess is that you confused two definitions from different papers.

In the paper, the attention module has weights $\alpha$ and the values to be weighted $h$, where the weights are derived from the recurrent neural network outputs, as described by the equations you quoted, and on the figure from the paper reproduced below.

![enter image description here

Similar thing happens in the Transformer model from the Attention is all you need paper by Vaswani et al, where they do use "keys", "querys", and "values" ($Q$, $K$, $V$). Vaswani et al define the attention cell differently:

$$ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\Big(\frac{QK^T}{\sqrt{d_k}}\Big)V $$

What they also use is multi-head attention, where instead of a single value for each $Q$, $K$, $V$, they provide multiple such values.

enter image description here

Where in the Transformer model, the $Q$, $K$, $V$ values can either come from the same inputs in the encoder (bottom part of the figure below), or from different sources in the decoder (upper right part of the figure). This part is crucial for using this model in translation tasks.

enter image description here

In both papers, as described, the values that come as input to the attention layers are calculated from the outputs of the preceding layers of the network. Both paper define different ways of obtaining those values, since they use different definition of attention layer.

$\endgroup$
6
$\begingroup$

This is an add up of what is K and V and why the author use different parameter to represent K and V. Short answer is technically K and V can be different and there is a case where people use different values for K and V.

K and V can be different! Example Offered!

What are K and V? Are they the same?

The short answer is that they can be the same, but technically they do not need to be the same.

Briefly introduce K, V, Q but highly recommend the previous answers: In the Attention is all you need paper, this Q, K, V are first introduced. In that paper, generally(which means not self attention), the Q is the decoder embedding vector(the side we want), K is the encoder embedding vector(the side we are given), V is also the encoder embedding vector. And this attention mechanism is all about trying to find the relationship(weights) between the Q with all those Ks, then we can use these weights(freshly computed for each Q) to compute a new vector using Vs(which should related with Ks). If this is self attention: Q, V, K can even come from the same side -- eg. compute the relationship among the features in the encoding side between each other.(Why not show strong relation between itself? Projection.)

Case where they are the same: here in the Attention is all you need paper, they are the same before projection. Also in this transformer code tutorial, V and K is also the same before projection.

Case where K and V is not the same: In the paper End-to-End Object Detection Appendix A.1 Single head(this part is an introduction for multi head attention, you do not have to read the paper to figure out what this is about), they offer an intro to multi-head attention that is used in the Attention is All You Need papar, here they add some positional info to the K but not to the V in equation (7), which makes the K and the V here are not the same.

Hope this helps.


Edit: As recommended by @alelom, I put my very shallow and informal understand of K, Q, V here.

For me, informally, the Key, Value and Query are all features/embeddings. Though it actually depends on the implementation but commonly, Query is feature/embedding from the output side(eg. target language in translation). Key is feature/embedding from the input side(eg. source language in translation), and for Value, basing on what I read by far, it should certainly relate to / be derived from Key since the parameter in front of it is computed basing on relationship between K and Q, but it can be a feature that is based on K but being added some external information or being removed some information from the source(like some feature that is special for source but not helpful for the target)...

What I have read(very limited, and I cannot recall the complete list since it is already a year ago, but all these are the ones that I found helpful and impressive, and basically it is just a summary of what I referred above...):

$\endgroup$
5
  • $\begingroup$ This answer is useful in making the point that K and V can be different but, like all other answers, fails to give a definition for V. $\endgroup$
    – alelom
    Jul 15, 2022 at 10:58
  • $\begingroup$ For me, informally, the Key, Value and Query are all features/embeddings. Though it actually depends on the implementation but commonly, Query is feature/embedding from the output side(eg. target language in translation). Key is feature/embedding from the input side(eg. source language in translation), and $\endgroup$
    – Flora Sun
    Jul 15, 2022 at 17:13
  • 1
    $\begingroup$ for Value, basing on what I read by far, it should certainly relate to / be derived from Key since the parameter in front of it is computed basing on relationship between K and Q, but it can be a feature that is based on K but being added some external information or being removed some information from the source(like some feature that is special for source but not helpful for the target)... $\endgroup$
    – Flora Sun
    Jul 15, 2022 at 17:13
  • $\begingroup$ Thank you! This is actually very helpful. Maybe you could embed this last comment in your answer, as it completes the OP Question (explaining Q, K and V). Also, if you have any source for your readings on V, if you could share them it would be great! $\endgroup$
    – alelom
    Jul 23, 2022 at 9:11
  • 1
    $\begingroup$ I edited the answer, copy and paste the comment into it. Although I am afraid that the reading list is not very helpful, it would be just a repeating list of what I mentioned in the original answer, since that is all I can recall now... The usage of V is actually from what I understood and generalized when I read in DETR they removed pos info from V but add it in Q. Just a very naive and untested idea. $\endgroup$
    – Flora Sun
    Jul 25, 2022 at 19:10
1
$\begingroup$

This is a concept that is the heart of the Transformer architecture and is difficult to explain or grasp.

We can say that in the process of learning the correct 'next' token to predict, three sets of weights per token are learned by backpropagating the loss - the Key, the Query and the Value weights. These form the base of the "Attention" mechanism. The video beautifully explains this at this location

The concept of Vector dot product is used to calculate the Value Vectors, which is the sum of the contribution of the dot product of Query and Key vectors. The intuition is that similar vectors in the Vector embedding space will have a larger dot product value and higher contribution. The Weights are then adjusted via Backpropagation which means that the learned weights represent a better Contextual Vector Embedding space for each token. ( Key and Query weights are multi-dimensional and there are multiple attention heads, so it is not one vector space but many)

In Transformers, there are multiple attention heads, so, which attention head to weightage more can also be tuned via weights. Basically Transformer network is this architecture, where the intuition of causal relationship between tokens is encoded as learnable weights in linear neural net layers.

I have tried to explain this in a simplified image below. It may not be fully accurate, but the overall high-level mechanism is right

Approximate working of a single attention head in Transformer Network

$\endgroup$
1
  • $\begingroup$ Thanks for the detailed answer. Unfortunately, it doesn't address my original question and is more of a general answer regarding the attention mechanism. $\endgroup$
    – Sean
    Aug 29, 2023 at 0:29
0
$\begingroup$

A lot of these posts are super technical, but I think make grasping the general idea difficult. So, I'm going to give a non-technical explanation. (If you want a technical explanation, this piece dives into more of the complexities.)

I'm going to use an analogy with dating apps. You have...

  1. A man who is trying to figure out how to improve his dating profile. (This man is the query.)
  2. A woman on the app. (This woman is the key.)
  3. That woman's ideal partner (someone she would want to match with immediately). (Her ideal partner is the value.)

Now, note that a dating profile is a representation of you. So, the man's goal is to improve his own representation.

The attention mechanism is a special app that can automatically improve the man's profile by taking into account who he wants to match with and turning his profile into a fusion of their ideal partners. The attention mechanism app works by learning to do the following:

  1. Given a particular man (the query), generate a new profile for that man that will be used to find women he might want to match with. (This function is called applying query weights.)
  2. Given a particular woman (the key), generate a new profile for that woman that will be used to find men that might want to match with her. (This function is called applying key weights.)
  3. Given the key woman's ideal partner (the value), generate a profile for that ideal partner. (This function is called applying value weights.)

Now, the attention mechanism app doesn't learn how to perform these tasks by being given the best-performing human-designed profiles and learning how to copy or imitate them. Instead, men will use the profiles from the app, and the results of whether they get more matches or not will help determine small tweaks the app can make to do better next time. (This is gradient descent.)

Given that these are the three things the attention app can improve at, here's how the app goes about coming up with suggestions for an improved profile for a given man:

  • Step 1. Generate a new profile for the man that will be used to find women he might want to match with. (Apply the query weights to the query.)
  • Step 2. For every woman on the app, generate a profile for them that will be used to find men that might want to match with her. (Apply the key weights to each key.)
  • Step 3. Compare the man's generated profile to each of the women's generated profiles. We assume that the more similar they are to each other, the more the man likely wants to match with her, so it's really important that step 1 and 2 are done well. (Generate attention scores.)
  • Step 4. Given the ideal partner for every woman on the app, generate a profile for each of the ideal partners. (Apply the value weights to each value to get a new value representation.)
  • Step 5. Take all the ideal profiles from step 4 and combine them into a new final profile suggestion for the man, but weight how much each woman's ideal partner profile influences the man's final suggested profile based on how likely it is that the man wants to match with her. (Output a weighted sum of the new value representations based on the attention scores.)

And that's it! That's how the attention mechanism works.

Now, you may be thinking "this is a great analogy, but how is attention applied in practice?" Great question! I come from a natural language processing background, so I'll explain how it's used in my field.

Those of us in natural language processing work with text, and so for us, queries, keys, and values are all words (technically tokens). We use attention to come up with better representations for the query words based on the key/value words. Often this is used in translation systems where the (query) word in the translated language may be a translation of multiple (key/value) words in the input language. The process of creating male and female profiles with similarities to see who the man wants to appeal to is analogous to creating new query and key vectors via the learned weights to represent the query and key/value words such that vectors pointing in the same direction indicate that the translation query word should be heavily impacted by that input key/value word. Then, just like how the suggested male profile was a weighted sum of all the generated ideal profiles, the representation of the translated query word becomes a weighted sum of all of the generated (via the learned value weight matrix) value vectors of the key/value words.

Now, you may also be wondering "What's the difference between attention and self-attention?" Notice how men and women were split up in this example? It was only men (decoder tokens) who were trying to improve their profiles, and they were only trying to improve them for women (encoder tokens). Self-attention is when everyone is trying to improve their profiles for everyone (better representations for all input tokens)! It's the same technique, but now everyone is a key and everyone's ideal partner is a value. So now everyone can create better representations of themselves that help them do what they want to do.

Hopefully this helps explain how transformers work in a way that's a bit more easy to understand. I'm hopeful that having a model like this in your head will help a lot of the math to make more sense.

(A sidenote for those of you who are planning to dive into the math: depending on the text, "query", "key", and "value" can either refer to the original encodings, the learned weight matrices that multiply the encodings, or the value of the original encodings multiplied by the weights. Usually $Q$, $K$ and $V$ either refer to the original encodings (as I have done here) or the encodings multiplied by the weights (as the transformer paper does), and $W^Q$, $W^K$, $W^V$ refer to the learned weight matrices, but you often have to figure it out by context. There's also some tricks that people use like normalization and heads etc., but you can also find those details explained well here.)

(And last, but certainly not least, my sincere apologies if it hurts (personally or systematically) to read yet another pretty classicly cishet example as an explainer; I unfortunately decided to prioritize this explanation's clarity over its inclusivity by including these gender and preference stereotypes to help readers keep track of the analogy. If you have a suggestion of how to improve the inclusivity without degrading the clarity, I would greatly appreciate it. Or if you just want to indicate that I should do better, a downvote will do that just fine. I'm not happy about the tradeoff either.)

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.