126

The key/value/query formulation of attention is from the paper Attention Is All You Need. How should one understand the queries, keys, and values The key/value/query concepts come from retrieval systems. For example, when you type a query to search for some video on Youtube, the search engine will map your query against a set of keys (video title, ...


29

I was also puzzled by the keys, queries, and values in the attention mechanisms for a while. After searching on the Web and digesting relevant information, I have a clear picture about how the keys, queries, and values work and why they would work! Let's see how they work, followed by why they work. In a seq2seq model, we encode the input sequence to a ...


24

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights). Skip connections form conceptually a '...


14

See Attention is all you need - masterclass, from 15:46 onwards Lukasz Kaiser explains what q, K and V are. So basically: q = the vector representing a word K and V = your memory, thus all the words that have been generated before. Note that K and V can be the same (but don't have to). So what you do with attention is that you take your current query (...


9

Where are people getting the key, query, and value from these equations? The paper you refer to does not use such terminology as "key", "query", or "value", so it is not clear what you mean in here. There is no single definition of "attention" for neural networks, so my guess is that you confused two definitions from different papers. In the paper, the ...


9

Tensorflow and Keras just expanded on their documentation for the Attention and AdditiveAttention layers. Here is a sneaky peek from the docs: The meaning of query, value and key depend on the application. In the case of text similarity, for example, query is the sequence embeddings of the first piece of text and value is the sequence embeddings of the ...


7

In positional encoding you encode the dimension with different frequency waves. Together with a position (on this wave) this gives you encoding that corresponds to each input. The encoding is subsequently added to the input. This procedure alters the angle between two embedding vectors. Suppose your word is embedded with a vector: $e_1,e_2,\dots ,e_d$. If ...


6

I'm going to try provide an English text example. The following is based solely on my intuitive understanding of the paper 'Attention is all you need'. Say you have a sentence: I like Natural Language Processing , a lot ! Assume that we already have input word vectors for all the 9 tokens in the previous sentence. So, 9 input word vectors. Looking at the ...


4

Queries is a set of vectors you want to calculate attention for. Keys is a set of vectors you want to calculate attention against. As a result of dot product multiplication you'll get set of weights a (also vectors) showing how attended each query against Keys. Then you multiply it by Values to get resulting set of vectors. Now let's look at word processing ...


4

By linear projections, one projection per attention head later used in the encoder-decoder attention. To be consistent with the notation in the paper, it would better to say that in the encoder-decoder attention $K = V$ which are the final states of the encoder and $Q$ are decoder states from a particular layer. They are used as input of the $\text{...


3

There are quite a few papers on this topic. The recent attempts to use pre-trained language models in MT are for instance: On the use of BERT for Neural Machine Translation Pre-trained Language Model Representations for Language Generation Back-translation is now considered a little bit tricky. Several recent papers showed (e.g. Domain, Translationese and ...


3

With respect to Deep Residual Learning for Image Recognition, I think it's correct to say that a ResNet contains both residual connections and skip connections, and that they are not the same thing. Here's a quotation from the paper: We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the ...


3

Teacher forcing effectively means that instead of using the predictions of your neural network at time step t (i.e the output of your RNN), you are using the ground truth. "Why does teacher forcing speed up training?" Because if you don't use teacher forcing, it is autoregressive, meaning that you need to calculate the labels first before passing ...


2

I think it's still very much an open question of which distance metrics to use for word2vec when defining "similar" words. Cosine similarity is quite nice because it implicitly assumes our word vectors are normalized so that they all sit on the unit ball, in which case it's a natural distance (the angle) between any two. As well, words that are similar tend ...


2

Do you mean by "ID scalar" really one number, so 1: Dog, 2: Cat, or a one hot encoded vector, so {1,0} Dog, {0,1} Cat. First Approach has the downside, that you already imply a n order/distance by construction, so for example if you encode: 1:Dog, 2:Tea, 3:Cat, you say that Dog and Tea are more simliar than Dog and Cat. So the first improvement is the one ...


2

For better and deeper understanding of the Residual Connection concept, you may want to also read this paper: Deep Residual Learning for Image Recognition. This is the same paper that is also referenced by "Attention Is All You Need" paper when explaining encoder element in the Transformers architecture.


2

Consider the following sentence in english and german (taken from wikipedia): He has eaten an apple. Er hat einen Apfel gegessen. The literal translation of the german sentence is He has an apple eaten. Suppose you're training an english -> german translation network. Suppose the network has already generated the first 3 words Er hat einen ...


2

1) Multi-head attention can be evaluated in parallel. It could simplify multi GPUs implementation. Single head attention could link it as you said, however, authors of the paper noticed that application of the multi-head attention is 'beneficial'. It's a black-box statement. 2) Feed-forward layers are always adding a "space" for a preparing mixture of ...


2

A transformer is a generative model. The goal of all generative models is to learn the probability distribution of the data (in this case, natural language sentences). To say "the model is fed the target sentence" isn't technically wrong, but it can be confusing. Instead, think of a generative model as a box with the following abilities*: You can ...


2

The reason is not that computing $p(e|f)$ would not be possible. The reason for the factorization is that you want to bring in the language mode $p(e)$ that can be used in decoding. At the time, SMT was invented, $n$-gram language models were quite good and it would be a pity not to use them. The probability $p(f|e)$ is factorized over phrases (in fact ...


2

I think I acquired some insights into this question after posting it 1.5 months ago, and since there are no other answers, I'll share them: Plain RNNs are, in practice, incapable of learning long-term dependencies, and while LSTMs can do it, they are still focused on recent inputs. This suits LMs just fine, because LMs are evaluated via PPL and similar ...


1

It is not really important in the encoder, but it plays a crucial role in the decoder. At the training time, you use the target sentence in the following way (with a 5-token sentence $w_1, \ldots, w_5$): [BOS] w₁ w₂ w₃ w₄ w₆ ↓ ↓ ↓ ↓ ↓ ↓ ┌─────────────────────────────┐ │ DECODER │ └─────────────────────────────┘ ...


1

The embeddings are trained jointly with the rest of the network. In the beginning, the embeddings are initialized randomly and the error gets back-propagated through the entire network down to the embeddings. When you train the embeddings jointly with the rest of the model, the problem often is that embeddings of the rare words only get updated once in a ...


1

Yes, this is the idea that the original paper promoted. Note, however, that it is a little bit tricky to use the term alignment. For purposes of the statistical machine translation, it was defined as a meaning correspondence of the source and target words. A highly cited 2017 study shows that the attention might learn very unintuitive alignments which seem ...


1

Machine translation cannot be easily evaluated by measuring classification accuracy. For every source sentence, there are very many valid translations, but your reference sentence is only one of the many possible correct ones. Evaluation of machine translation is still an open problem. There is even an annual competition where teams try to develop a metric ...


1

as I understand, BPE is just used for word segmentation (compress size of the dictionary) so how will it produce a numeric vector? BPE(an illustration can be found here) (and unigram language model) is a subword algorithm, and it just tokenizes the text into subwords which can be just treated as words separated by spaces because they will be transformed ...


1

Generally, machine translation is translating a sentence from an original language to a target language. To be more precise, in the context of statistical or neural machine translation, the goal is to Model the conditional distribution $p(y|x)$ where $x$ is the source sentence and $y$ the target. At test time, sample a high quality result from this ...


1

I would not go for an artificial dataset because they are too easy for a network to learn and you might not discover some bugs. It happened to me that I had a code that was able to do trivial tasks like capitalization but failed to scale to more complex tasks because of a bug in the model architecture. An easy sequence-to-sequence task is transliteration. ...


1

Forget about V Focus on what the objective of MatMul is in the Scaled dot product attention using Q and K. Attention For the sentence "jane visits africa". When your eyes see jane, your brain looks for the most related word in the rest of the sentence to understand what jane is about (query). Your brain focuses or attends to the word visit (key). ...


1

A sequence to sequence model produces a probability distribution over sequences $Y$ conditioned on another sequence $X$. At test time, you may want to find the mode of that distribution: $\max_Y P(Y|X;\theta)$ -- the sequence which is most likely. For example if you have a translation model, you want to find the translation which has the highest ...


Only top voted, non community-wiki answers of a minimum length are eligible