# Matrices K and V in the decoder part in the transformer model?

There is something I do not get in the illustrated transformer article from Jay Alamar (http://jalammar.github.io/illustrated-transformer/). In the decoder side paragraph, he said

The encoder start by processing the input sequence. The output of the top encoder is then transformed into a set of attention vectors K and V.

How the hell do we compute those K and V?. If i well understood, the output of the encoder is a matrice (Number of words x Embedding Length). So where those K and V come from please? By linear projections, one projection per attention head later used in the encoder-decoder attention.

To be consistent with the notation in the paper, it would better to say that in the encoder-decoder attention $$K = V$$ which are the final states of the encoder and $$Q$$ are decoder states from a particular layer. They are used as input of the $$\text{MultiHead}$$ (unnumbered equation on top of page 5) function where they are projected for individual heads. So in the individual head computation according to Equation 1: it is indeed true $$K$$ and $$V$$ are linear projections of the final encoder states (outputs of the top encoder layer).

• Finally I think I understand the whole thing. Thank you Sir! Very nice from you! Mar 16 '20 at 14:27