1
$\begingroup$

(For this example, let's say we're using a Transformer to translate from English to French.)

For the decoder in a Transformer, the second attention layer takes K and V from the encoder and then takes Q from the decoder's previous attention layer.

First of all, how do K and V get from the encoder to the decoder? Is the encoder's output copied twice and then directly fed into the decoder? Or is it transformed to K and V with their respective weight matrices?

Second, why are K and V the same? I understand that Q is supposed to be the resulting French sentence and that K tells the model how to map the English sentence to French. But why is V the same as K? The only explanation I can think of is that V's dimensions match the product of Q & K. However, V has K's embeddings, and not Q's. I'm assuming Q and K & V have different embedding values because they are two different languages.

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, in the decoder, the queries come from the decoder's hidden states, and the keys and values come from the encoder's hidden states. This does not necessarily mean that $K=V$.

It would be true only if the dot-product attention were applied directly. However, Transformers use multi-head attention with a linear transformation from the states to the queries, keys, and values. This means that keys and values are different linear projections of the same hidden states. A usual interpretation of the linear projections is extracting relevant information for kyes and values.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for the clarification. However, I am still confused about why V is derived from the encoder's hidden state, i.e. from the input, rather than from the decoder's previous attention layer, i.e from the output. After all, the output is what we are trying to, well, output. $\endgroup$ Commented Jan 13, 2023 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.