First, understand Q and K
First, focus on the objective of First MatMul in the Scaled dot product attention using Q and K.
Intuition on what is Attention
For the sentence "jane visits africa".
When your eyes see jane, your brain looks for the most related word in the rest of the sentence to understand what jane is about (query). Your brain focuses or attends to the word visit (key).
This process happens for each word in the sentence as your eyes progress through the sentence.
First MatMul as Inquiry System using Vector Similarity
The first MatMul implements an inquiry system or question-answer system that imitates this brain function, using Vector Similarity Calculation. Watch CS480/680 Lecture 19: Attention and Transformer Networks by professor Pascal Poupart to understand further.
Think about the attention essentially being some form of approximation of SELECT that you would do in the database.
Think of the MatMul as an inquiry system processes the inquiry: "For the word q that your eyes see in the given sentence, what is the most related word k in the sentence to understand what q is about?" The inquiry system provides the answer as the probability.
| q | k | probability |
|---|---|---|
| jane | visit | 0.94 |
| visit | africa | 0.86 |
| africa | visit | 0.76 |
Note that the softmax is used to scale (in yellow) to normalize values into probabilities so that their sum becomes 1.0.
There are multiple ways to calculate the similarity between vectors such as cosine similarity. Transformer attention uses simple dot product.
Where are Q and K are from
The transformer encoder training builds the weight parameter matrices WQ and Wk in the way Q and K builds the Inquiry System that answers the inquiry "What is k for the word q".
The calculation goes like below where x is a sequence of position-encoded word embedding vectors that represents an input sentence.
Picks up a word vector (position encoded) from the input sentence sequence, and transfer it to a vector space Q. This becomes the query.
$Q = X \cdot W_{Q}^T$Pick all the words in the sentence and transfer them to the vector space K. They become keys and each of them is used as key.
$K = X \cdot W_K^T$For each (q, k) pair, their relation strength is calculated using dot product.
$q\_to\_k\_similarity\_scores = matmul(Q, K^T)$Weight matrices $W_Q$ and $W_K$ are trained via the back propagations during the Transformer training.
We first needs to understand this part that involves Q and K before moving to V.
Then, understand how V is created using Q and K
Second Matmul
Self Attention then generates the embedding vector called attention value as a bag of words where each word contributes proportionally according to its relationship strength to q. This occurs for each q from the sentence sequence. The embedding vector is encoding the relations from q to all the words in the sentence.
References
There are multiple concepts that will help understand how the self attention in transformer works, e.g. embedding to group similars in a vector space, data retrieval to answer query Q using the neural network and vector similarity.
Transformers Explained Visually (Part 2): How it works, step-by-step give in-detail explanation of what the Transformer is doing.
CS480/680 Lecture 19: Attention and Transformer Networks - This is probably the best explanation I found that actually explains the attention mechanism from the database perspective.
Illustrated Guide to Transformers Neural Network: A step by step explanation
Distributed Representations of Words and Phrases and their Compositionality - It helps understand how word2vec works to group/categorize words in a vector space by pulling similar words together, and pushing away non-similar words using negative sampling.
Generalized End-to-End Loss for Speaker Verification - Continuation to understand embedding to pull together siimilars and pushing away non-similars in a vector space.
Transformer model for language understanding - TensorFlow implementation of transformer
The Annotated Transformer - PyTorch implementation of Transformer
Update
Getting meaning from text: self-attention step-by-step video has visual representation of query, key, value.
