3
$\begingroup$

My understanding is for translation task K should be the same with V, but in Transformer K and V are generated by two different(randomly initialized) matrix $W^K, W^V$, therefore not the same. Can any one tell me why?

$\endgroup$
  • $\begingroup$ why do you think k should be the same as v? $\endgroup$ – shimao Oct 14 '19 at 2:46
  • $\begingroup$ We compute similarity of Q and K and use its result as weight to pass K information into the new value of Q, V and K should be the same so that Q know how much K's information should be added to Q $\endgroup$ – eric2323223 Oct 14 '19 at 6:09
  • $\begingroup$ does this answer help? stats.stackexchange.com/a/424127/95569 $\endgroup$ – dontloo Oct 16 '19 at 2:45
1
$\begingroup$

I guess the reason why the specific terms "query", "key" and "value" were chosen is that this attention mechanism resembles a memory access mechanism. The query is the specific element for which we seek a representation. The role of the keys is to respond more or less to the query and the values are here to compose an answer. Keys and values are necessarily related but do not play the same roles.

For example, given the word query "network", you might want the key words "neural" and "social" to generate high weights since "neural network" and "social network" are common terms. It means that the dot products between the query and these two keys are high and thus the two key vectors are similar. Nevertheless, the values for "neural" and "social" should be dissimilar since they don't deal with the same topic. Using the same representation for keys and values doesn't allow this.

Somehow, using the same transformation for keys and values might still work, but you'll lose a lot of expressiveness and might need much more parameters to achieve similar performances.

EDIT: I just found a better explanation of the query, key and value terms in this post.

$\endgroup$
  • $\begingroup$ Could you please explain more about "you'll lose a lot of expressiveness and might need much more parameters to achieve similar performances" ? $\endgroup$ – eric2323223 Oct 22 '19 at 1:35
  • $\begingroup$ 1) It would mean that you use the same matrix for K and V, therefore you lose 1/3 of the parameters which will decrease the capacity of the model to learn. 2) As I explain in the second paragraph, by forcing K and V to play the same role, you lose the capacity of the model to distinguish between key interaction and value composition. $\endgroup$ – Robin Oct 22 '19 at 13:53
  • $\begingroup$ I am still confused why use K, Q, V instead of just K, Q, it make no sense to me K, V are different in language translation context. $\endgroup$ – eric2323223 Oct 23 '19 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.