0
$\begingroup$

There is something I do not get in the illustrated transformer article from Jay Alamar (http://jalammar.github.io/illustrated-transformer/). In the decoder side paragraph, he said

The encoder start by processing the input sequence. The output of the top encoder is then transformed into a set of attention vectors K and V.

How the hell do we compute those K and V?. If i well understood, the output of the encoder is a matrice (Number of words x Embedding Length). So where those K and V come from please?

enter image description here

$\endgroup$
4
$\begingroup$

By linear projections, one projection per attention head later used in the encoder-decoder attention.

To be consistent with the notation in the paper, it would better to say that in the encoder-decoder attention $K = V$ which are the final states of the encoder and $Q$ are decoder states from a particular layer. They are used as input of the $\text{MultiHead}$ (unnumbered equation on top of page 5) function where they are projected for individual heads.

enter image description here

So in the individual head computation according to Equation 1:

enter image description here

it is indeed true $K$ and $V$ are linear projections of the final encoder states (outputs of the top encoder layer).

$\endgroup$
1
  • $\begingroup$ Finally I think I understand the whole thing. Thank you Sir! Very nice from you! $\endgroup$
    – hans glick
    Mar 16 '20 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.