4
$\begingroup$

My understanding is for translation task K should be the same with V, but in Transformer K and V are generated by two different(randomly initialized) matrix $W^K, W^V$, therefore not the same. Can any one tell me why?

$\endgroup$
3
  • $\begingroup$ why do you think k should be the same as v? $\endgroup$
    – shimao
    Oct 14 '19 at 2:46
  • $\begingroup$ We compute similarity of Q and K and use its result as weight to pass K information into the new value of Q, V and K should be the same so that Q know how much K's information should be added to Q $\endgroup$ Oct 14 '19 at 6:09
  • $\begingroup$ does this answer help? stats.stackexchange.com/a/424127/95569 $\endgroup$
    – dontloo
    Oct 16 '19 at 2:45
2
$\begingroup$

I guess the reason why the specific terms "query", "key" and "value" were chosen is that this attention mechanism resembles a memory access mechanism. The query is the specific element for which we seek a representation. The role of the keys is to respond more or less to the query and the values are here to compose an answer. Keys and values are necessarily related but do not play the same roles.

For example, given the word query "network", you might want the key words "neural" and "social" to generate high weights since "neural network" and "social network" are common terms. It means that the dot products between the query and these two keys are high and thus the two key vectors are similar. Nevertheless, the values for "neural" and "social" should be dissimilar since they don't deal with the same topic. Using the same representation for keys and values doesn't allow this.

Somehow, using the same transformation for keys and values might still work, but you'll lose a lot of expressiveness and might need much more parameters to achieve similar performances.

EDIT: I just found a better explanation of the query, key and value terms in this post.

$\endgroup$
5
  • $\begingroup$ Could you please explain more about "you'll lose a lot of expressiveness and might need much more parameters to achieve similar performances" ? $\endgroup$ Oct 22 '19 at 1:35
  • $\begingroup$ 1) It would mean that you use the same matrix for K and V, therefore you lose 1/3 of the parameters which will decrease the capacity of the model to learn. 2) As I explain in the second paragraph, by forcing K and V to play the same role, you lose the capacity of the model to distinguish between key interaction and value composition. $\endgroup$
    – Robin
    Oct 22 '19 at 13:53
  • $\begingroup$ I am still confused why use K, Q, V instead of just K, Q, it make no sense to me K, V are different in language translation context. $\endgroup$ Oct 23 '19 at 9:24
  • 1
    $\begingroup$ @eric2323223 Keys and values are different since it allows the network to separate unidirectional relationship between input tokens from bi-directional relationships. For example Robin's example in his answer: The token "Network" can be related to "neural" and "social" in some very similar ways, but the "answer" from the respective relationships is not neccessary equally similar. So both "neural" and "social" tokens can respond to the same Query coming from the "Network"-token, i.e. the keys for "neural" and "social" is very similar, but the values is returned is not the same for both tokens. $\endgroup$
    – Plarsen
    Sep 3 '20 at 10:59
  • 1
    $\begingroup$ @eric2323223 I try re-phrase my comment: The Query->Key connection answers the question "How is this token (Network) related to this token (Social)?" and then the returned result (partly based on the Value) represents "What is the result of this relationship?". So the keys is like meta-labels, listeners that is triggered and responsive to the queries while values is used when responding to the query - it's not sending the pure meta-labels, the triggers, to the next attention-layer. $\endgroup$
    – Plarsen
    Sep 3 '20 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.