My understanding is for translation task K should be the same with V, but in Transformer K and V are generated by two different(randomly initialized) matrix $W^K, W^V$, therefore not the same. Can any one tell me why?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ why do you think k should be the same as v? $\endgroup$– shimaoOct 14, 2019 at 2:46
-
$\begingroup$ We compute similarity of Q and K and use its result as weight to pass K information into the new value of Q, V and K should be the same so that Q know how much K's information should be added to Q $\endgroup$– eric2323223Oct 14, 2019 at 6:09
-
$\begingroup$ does this answer help? stats.stackexchange.com/a/424127/95569 $\endgroup$– dontlooOct 16, 2019 at 2:45
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
I guess the reason why the specific terms "query", "key" and "value" were chosen is that this attention mechanism resembles a memory access mechanism. The query is the specific element for which we seek a representation. The role of the keys is to respond more or less to the query and the values are here to compose an answer. Keys and values are necessarily related but do not play the same roles.
For example, given the word query "network", you might want the key words "neural" and "social" to generate high weights since "neural network" and "social network" are common terms. It means that the dot products between the query and these two keys are high and thus the two key vectors are similar. Nevertheless, the values for "neural" and "social" should be dissimilar since they don't deal with the same topic. Using the same representation for keys and values doesn't allow this.
Somehow, using the same transformation for keys and values might still work, but you'll lose a lot of expressiveness and might need much more parameters to achieve similar performances.
EDIT: I just found a better explanation of the query, key and value terms in this post.
-
$\begingroup$ Could you please explain more about "you'll lose a lot of expressiveness and might need much more parameters to achieve similar performances" ? $\endgroup$ Oct 22, 2019 at 1:35
-
$\begingroup$ 1) It would mean that you use the same matrix for K and V, therefore you lose 1/3 of the parameters which will decrease the capacity of the model to learn. 2) As I explain in the second paragraph, by forcing K and V to play the same role, you lose the capacity of the model to distinguish between key interaction and value composition. $\endgroup$– RobinOct 22, 2019 at 13:53
-
$\begingroup$ I am still confused why use K, Q, V instead of just K, Q, it make no sense to me K, V are different in language translation context. $\endgroup$ Oct 23, 2019 at 9:24
-
2$\begingroup$ @eric2323223 Keys and values are different since it allows the network to separate unidirectional relationship between input tokens from bi-directional relationships. For example Robin's example in his answer: The token "Network" can be related to "neural" and "social" in some very similar ways, but the "answer" from the respective relationships is not neccessary equally similar. So both "neural" and "social" tokens can respond to the same Query coming from the "Network"-token, i.e. the keys for "neural" and "social" is very similar, but the values is returned is not the same for both tokens. $\endgroup$ Sep 3, 2020 at 10:59
-
1$\begingroup$ @eric2323223 I try re-phrase my comment: The Query->Key connection answers the question "How is this token (Network) related to this token (Social)?" and then the returned result (partly based on the Value) represents "What is the result of this relationship?". So the keys is like meta-labels, listeners that is triggered and responsive to the queries while values is used when responding to the query - it's not sending the pure meta-labels, the triggers, to the next attention-layer. $\endgroup$ Sep 3, 2020 at 12:50