6

There is nothing in the self-attention parameterization that would make it limited to a pre-defined length. The attention is done by a dot-product of all state-pairs and then as a weighted sum of the projected states. The transformer encoder uses position encoding. This is the only component that could be length-dependent, however, this is not part of the ...


5

I'll tell you what I know/have heard about it. There's probably other analysis out there I haven't seen. I think multi-head attention was introduced in the transformer network paper. Their justification/explanation is: Another paper on a modified Transformer architecture questions this understanding though:


4

The location address mechanism doesn't focus on a single address, but outputs a distribution on addresses to focus on. This is necessary because otherwise the process would be nondifferentiable, and gradients wouldn't be able to flow backward through the addressing mechanism, and you wouldn't be able to train it. However, if you repeatedly convolve these ...


3

I think I acquired some insights into this question after posting it 1.5 months ago, and since there are no other answers, I'll share them: Plain RNNs are, in practice, incapable of learning long-term dependencies, and while LSTMs can do it, they are still focused on recent inputs. This suits LMs just fine, because LMs are evaluated via PPL and similar ...


3

What do these weights learn? At the most literal level, they learn to predict the tokens of a sequence of text, given a corrupted form of that same sequence. In other words, they learn common patterns in language. BERT is a gigantic pseudo–language model. Are weights learned for each token? It’s easy for a misunderstanding to arise here. For the same word ...


2

The original "Attention is all you need" paper use sine positional encoding. You can find a great in-depth explanation on this topic by Jonathan Kernes here: https://towardsdatascience.com/master-positional-encoding-part-i-63c05d90a0c3. While positional embedding is basically a learned positional encoding. Hope that it helps!


2

Because Transformers are black-box models, it is hard to say, what the keys and values really are, but the motivation is that might want to retrieve something else than what is your search criterion. Imagine something like SQL-like query: get phone numbers of people that have a similar name to "Jindrich". "Jindrich" is a query, the ...


2

RNN's operate on the input sequence one at a time going down the line. Transformers have input width greater than the length of the longest input sequence. It eats up the whole sequence at once, chews it through the different attention layers, then spits it out. So it can attend to anywhere in the input at any time, but this means you can't run a given model ...


2

A transformer is a generative model. The goal of all generative models is to learn the probability distribution of the data (in this case, natural language sentences). To say "the model is fed the target sentence" isn't technically wrong, but it can be confusing. Instead, think of a generative model as a box with the following abilities*: You can ...


2

The weight matrices are $n$ by $m$ with $n >> m$. So $W_Q W_K^T$ is not just any matrix, it's $n$ by $n$ but with rank only $m$ -- there are fewer parameters, and computing $QK^T$ is much faster than $X W' X^T$ for some full rank $W'$


1

Are the attention weights based solely on position of the tokens? No, they're based on the specific values that appear in the sequence. The attention formula actually has no information about position at all. Because in natural language processing we believe the order of words is important, we will often incorporate a positional encoding into the vectors we ...


1

The different words in a sentence can relate to each other in many different ways simultaneously. For example, distinct syntactic, semantic, and discourse relationships can hold between verbs and their arguments in a sentence. It would be difficult for a single transformer block to learn to capture all of the different kinds of parallel relations among its ...


1

The mask is typically a square matrix with a upper-right triangle of ones (or True in the pytorch implementation), and the rest is filled with zeros. 0 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 The ones will mask/put to zero the attention coefficients on those positions (read below). The shape of this matrix, a upper-right triangle, allows the transformer to avoid a ...


1

This is already partially answered in the comment. With the standard attention, you basically multiply $V$ by a vector (or matrix) of probabilities $A$, so that you pay more (higher probabilities) or less (lower probabilities) "attention" to particular values of $V$ $$ \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V = A V $$ What you seem to ...


1

There is no need separately consider such cases. In fact, in machine translation, it is mostly the case that the source and target sentences have different lengths. The encoder states are the keys and values of the attention (in the simples case of single-head Bahdanau's attention there are literally the same). In every decoder step, you have one query ...


1

It doesn't. Positional encoding is not part of the multi-head attention layer.


1

You definitely need the masking for the padded positions in the encoder. Nothing changes there. If you implement the decoder correctly/efficiently, you should not need the triangle mask in the decoder either. In an efficient implementation, you only have one query that comes from the most recently added token, and you do the self-attention with the hidden ...


1

The sublayers refer to the self/cross multi-head attention layers, as well as the position-wise feedfoward networks. Your code is mostly correct, but: your pseudocode accidentally overwrites the value of the original x. The layer norm is applied after the residual addition. there's no ReLU in the transformer (other than within the position-wise feed-forward ...


1

The semicolon operator in the formulas actually denotes concatenation and it the concatenation that they refer to in the paper (opposed to dot product). The summation in Bahdanau's formulation with the sum of two projections is equivalent to a projection of the vector concatenation ($\oplus$ denotes concatenation): $$W_a s_{i-j} + U_a h_j = (W_a \oplus U_a) \...


1

BERT just need the encoder part of the Transformer, this is true but the concept of masking is different than the Transformer. You mask just a single word (token). So it will provide you the way to spell check your text for instance by predicting if the word is more relevant than the wrd in the next sentence. My next <mask> will be different. The GPT-...


1

my quesion is: for the decoder, doesn't this linear layer mess up with the masking of the attention? You know already that the answer is NO, because masked self attention goes before multi head attention with three inputs. Indeed these linear weights will learn the dependencies among all the tokens and during inference could there be a problem, maybe(?). ...


1

No, the decoder cannot generate the outputs in parallel. It generates the output autoregressively, i.e., the probability distribution of the $i+1$-th token is conditioned on the $i$-th output token. At the training time, this is simulated by the triangle maks in the self-attention that prevents the decoder from attending to the tokens that will be generated ...


1

Here is what you may need to read: Where do we get these positional embeddings? A simple and effective approach is to start with randomly initialized embeddings corresponding to each possible input position up to some maximum length. For example, just as we have an embedding for the word fish, we’ll have an embedding for the position 3. As with word ...


1

It would be a good idea to employ adaptive batching, that is you pad a batch(append zeros) by the longest case in that and in the reference mode the batch size is just one and the length is the length of that case. You don't need bucketing but enjoy the speeding up. You can do that by first padding all cases in the data pipeline by the max length and second ...


1

The feed-forward layer is weights that is trained during training and the exact same matrix is applied to each respective token position. Since it is applied without any communcation with or inference by other token positions it is a highly parallelizable part of the model. The role and purpose is to process the output from one attention layer in a way to ...


1

A model that takes any type of input (presented as a vector, matrix or tensor) and summarises the input into natural language? There are for sure models which do image captioning, video captioning, text summarization, etc. You can of course write some wrapper code which runs one of many models depending on the input modality, enabling it to summarize ...


1

The embeddings are trained jointly with the rest of the network. In the beginning, the embeddings are initialized randomly and the error gets back-propagated through the entire network down to the embeddings. When you train the embeddings jointly with the rest of the model, the problem often is that embeddings of the rare words only get updated once in a ...


1

The positional encoding is a static function that maps an integer inputs to real-valued vectors in a way that captures the inherent relationships among the positions. That is, it captures the fact that position 4 in an input is more closely related to position 5 than it is to position 17. While for the position embedding there will be plenty of training ...


1

as I understand, BPE is just used for word segmentation (compress size of the dictionary) so how will it produce a numeric vector? BPE(an illustration can be found here) (and unigram language model) is a subword algorithm, and it just tokenizes the text into subwords which can be just treated as words separated by spaces because they will be transformed ...


1

Transformers require quadratic memory with the length of the text, using too long texts might result in memory issues. E.g., in Huggingface's implementation, most of the models do not accept sequences longer than 512 subwords. Making pre-trained Transformers work efficiently for long texts is an active research area, you can have a look at a paper called ...


Only top voted, non community-wiki answers of a minimum length are eligible