14
$\begingroup$

My understanding is for translation task K should be the same with V, but in Transformer K and V are generated by two different(randomly initialized) matrix $W^K, W^V$, therefore not the same. Can any one tell me why?

$\endgroup$
4
  • 1
    $\begingroup$ why do you think k should be the same as v? $\endgroup$
    – shimao
    Commented Oct 14, 2019 at 2:46
  • $\begingroup$ We compute similarity of Q and K and use its result as weight to pass K information into the new value of Q, V and K should be the same so that Q know how much K's information should be added to Q $\endgroup$ Commented Oct 14, 2019 at 6:09
  • $\begingroup$ does this answer help? stats.stackexchange.com/a/424127/95569 $\endgroup$
    – dontloo
    Commented Oct 16, 2019 at 2:45
  • $\begingroup$ This article goes into more depth (and this other answer of mine goes into less depth), but the general idea is that the key weights determine how to represent an item in a good way to figure out how much it should impact the query item. Then, once we've determined how big of an impact it should have (the attention score), we also need to figure out what that impact should look like, which is the job of the value weights. $\endgroup$
    – Pro Q
    Commented Feb 11 at 10:52

1 Answer 1

6
$\begingroup$

I guess the reason why the specific terms "query", "key" and "value" were chosen is that this attention mechanism resembles a memory access mechanism. The query is the specific element for which we seek a representation. The role of the keys is to respond more or less to the query and the values are here to compose an answer. Keys and values are necessarily related but do not play the same roles.

For example, given the word query "network", you might want the key words "neural" and "social" to generate high weights since "neural network" and "social network" are common terms. It means that the dot products between the query and these two keys are high and thus the two key vectors are similar. Nevertheless, the values for "neural" and "social" should be dissimilar since they don't deal with the same topic. Using the same representation for keys and values doesn't allow this.

Somehow, using the same transformation for keys and values might still work, but you'll lose a lot of expressiveness and might need much more parameters to achieve similar performances.

EDIT: I just found a better explanation of the query, key and value terms in this post.

$\endgroup$
5
  • $\begingroup$ Could you please explain more about "you'll lose a lot of expressiveness and might need much more parameters to achieve similar performances" ? $\endgroup$ Commented Oct 22, 2019 at 1:35
  • $\begingroup$ 1) It would mean that you use the same matrix for K and V, therefore you lose 1/3 of the parameters which will decrease the capacity of the model to learn. 2) As I explain in the second paragraph, by forcing K and V to play the same role, you lose the capacity of the model to distinguish between key interaction and value composition. $\endgroup$
    – Robin
    Commented Oct 22, 2019 at 13:53
  • $\begingroup$ I am still confused why use K, Q, V instead of just K, Q, it make no sense to me K, V are different in language translation context. $\endgroup$ Commented Oct 23, 2019 at 9:24
  • 3
    $\begingroup$ @eric2323223 Keys and values are different since it allows the network to separate unidirectional relationship between input tokens from bi-directional relationships. For example Robin's example in his answer: The token "Network" can be related to "neural" and "social" in some very similar ways, but the "answer" from the respective relationships is not neccessary equally similar. So both "neural" and "social" tokens can respond to the same Query coming from the "Network"-token, i.e. the keys for "neural" and "social" is very similar, but the values is returned is not the same for both tokens. $\endgroup$ Commented Sep 3, 2020 at 10:59
  • 1
    $\begingroup$ @eric2323223 I try re-phrase my comment: The Query->Key connection answers the question "How is this token (Network) related to this token (Social)?" and then the returned result (partly based on the Value) represents "What is the result of this relationship?". So the keys is like meta-labels, listeners that is triggered and responsive to the queries while values is used when responding to the query - it's not sending the pure meta-labels, the triggers, to the next attention-layer. $\endgroup$ Commented Sep 3, 2020 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.