6
$\begingroup$

In the paper Attention Is All You Need the matrix of outputs is computed as follows:

enter image description here.
In the blog post The Illustrated Transformer it says that the matrices were trained during the process.

So for each word, we create a Query vector, a Key vector, and a Value vector. These vectors are created by multiplying the embedding by three matrices that we trained during the training process

It is just not clear where do we get the WQ,WK and WV matrices that are used to create Q,K,V. All the resources explaining the model mention them if they are already pre-trained somewhere.enter image description here

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
2
$\begingroup$

K,Q and V are representations of encoder and decoder states and vary according to problem. The weight matrices correspond to the linear transformation of the states.They are trained as a part of the whole neural network block. This blog might be helpful. https://medium.com/lsc-psd/introduction-of-self-attention-layer-in-transformer-fc7bff63f3bc

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you! That is exactly what I was looking for. $\endgroup$
    – spacer.34
    Commented May 26, 2020 at 10:10
  • $\begingroup$ So where Wq Wk Wv comes from? I did not saw a explanation in the link. $\endgroup$
    – Wuaner
    Commented Oct 6 at 1:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.