I am reading the paper "Attention is all you need" (https://arxiv.org/abs/1706.03762). In transformer architecture, we have 3 vectors(key,value and query) for each word. I don't understand the need of value vector. Why won't just the dot product of query and key vector suffice? Dot product of query and key vector would give us how closely are the words related to each other which is exactly what we need in self-attention layer. What is the need to scale this value with value vector?
Thanks.
Dot product of query and key vector would give us how closely are the words related to each other which is exactly what we need in self-attention layer.
Well, we want the layer to output more than a scalar score. We want it to output a vector of "features". For example, if I type a query into a search engine and it tells me that the score of the best-matching webpage is 0.94... well that doesn't really help, I want the actual content of the webpage.
There are attention mechanisms which combine the key and value vector in one, but they still output a vector.
The result from the Query <-> Key is close to a binary match/non-match between one token and another and then the value is a way to return some useful information from the matched token.
Jay Alammar's explaination in The illustrated transformer is one of the best in my opinion, especially step 1-6 in chapter Self-Attention in Detail: http://jalammar.github.io/illustrated-transformer/
Most documentation tend to quickly move into abstract perspectives and paper formulas but if you are like me you need atleast one simple example, without any formulas, to understand the basics, and from there the documentations makes more sense. I created a simple illustrative Excel-document a while ago with formulas (not descriptive ones, but practical) simulating the encoder part of one encoder layer. It's not working exactly as suggested in "Attention is all you need" but somewhat similar. It's extreamly small and practically useless (It's Excel after all): sequence length: 9, only 3 dimensions för Q,K,V, positional encoding is 2 dims, 4 attention heads. The vocabulary is only the 9 words/tokens in the example and all weights that is supposed to be trained in the model is just random numbers generated on the fly. The document lacks any training functionality. Oh, and it doesn't use softmax, so I guess the Query <-> Key result is far from binary. I didn't use dot product either since 3 of the 4 attention heads only query one single dimension.
Anyway, real examples (all the way through every detail) can sometimes work as an "ice breaker" for understanding.
Excel-document: https://artificial.se/AttentionDemo.xlsx
Screenshot:
(If someone improves the Excel-document, finish it with decoder or training function - I guess that will require macro-enable it - or anything else, then please share your version with the rest of us that need practical out-of-the-box easy-to-use examples to play around with.)
