Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
So attention and transformer models can be used for machine translation.
Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for example a word in the source language might be two words in the target language)
How do K, Q, V in the attention models work to deal with this problem? Are there several values for that corresponding key? (in this case, several words corresponding to one concept?)
There is no need separately consider such cases. In fact, in machine translation, it is mostly the case that the source and target sentences have different lengths.
The encoder states are the keys and values of the attention (in the simples case of single-head Bahdanau's attention there are literally the same). In every decoder step, you have one query vector (corresponding to a token that is about to be generated). In the original Bahdanau's paper, it is denoted as $s_i$.
This is used to query keys to get an attention distribution:
$$\alpha_i = \mathrm{softmax}_j a(s_i, h_j) $$
Where $a$ is the energy function (in Bahdanau's version, it is non-linear projection, in Luong's version, it is dot-product) a $h_j$ is the $j$-th decoder state. In other words, for one query vector $s_i$, we get a multinomial distribution with as many as the number of keys=number of encoder states.
The next step is computing the context vector. Here, the query is no longer relevant. We have one scalar number for each value vector, so we can compute the context vector
$$ c_i = \sum_j \alpha_{i,j} h_j $$
This is done independently for each decoder state, so it does not matter what the relation between the source and target length is.
