Questions tagged [attention]
Methods for aggregating a set feature vectors into a single feature vector relevant to a context.
91
questions
0
votes
0
answers
11
views
What does the weight matrix in dot-product attention capture about the training set?
Let $x, y \in \mathbb{R}^d$ be the input and output vectors in a seq2seq setup with one self-attention head. When we compute dot-product attention using queries and keys, given by, $q^Tk = x^TW_q^TW_k ...
- 153
0
votes
0
answers
23
views
How does attention add expressive power to encoder-decoders?
I am learning about the attention mechanism for the first time, and the context in which it has been introduced (watching Lecture 8 of Stanford's CS224N) is that of language translation using seq2seq ...
- 153
0
votes
1
answer
7
views
Does the attention mechanism (in CNNs) bring additional parameters/weights to learn to the network?
The idea of the attention mechanism is based on using some weighted sum of the output of some layers in deep networks. I see the process in forward propagation, and it seems that the attention ...
- 189
0
votes
0
answers
13
views
How to properly mask MultiHeadAttention for sliding window time series data
I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent ...
- 133
1
vote
1
answer
35
views
How to obtain Key, Value and Query in Attention and Multi-Head-Attention
I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and ...
- 121
0
votes
2
answers
10
views
Do value and key of additive attention need to have the same dimension?
For the documented tensorflow-keras implementation of additive attention, it is stated that the input tensors are:
query: Query Tensor of shape ...
- 101
3
votes
2
answers
41
views
About basic understanding of attention mechanism and model weights
In the image domain, for a given image, suppose that we want to understand if there is a bird on that image (0 and 1 label = no bird and bird), attention mechanism helps to pay more attention to the ...
- 189
0
votes
0
answers
13
views
Extract Local Attention Positions
I am working on a project, I want to extract the local attention positions present in the feature maps of every level in the FPN. Below is the visualization:
I am using MMDetection toolbox for ...
0
votes
1
answer
46
views
Trainable weights in attention mechanism
I am wondering what are the trainable weights inside an attention-powered transformer. I figure the feed-forward layer contain trainable weights and the token embeddings, but what other parts contain ...
6
votes
1
answer
810
views
Why are residual connections needed in transformer architectures?
Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training.
This problem is ...
- 177
1
vote
1
answer
21
views
Why the sequence of matrix transformations in The Annotated Transformer?
Forgive me the long introduction, but I want to be very specific with my question.
The Annotated Transformer implements the Transformer architecture from "Attention Is All You Need". The <...
- 11
2
votes
1
answer
50
views
Hard attention derivations
I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
- 47
1
vote
0
answers
25
views
How can one feed all of the input to an RNN, and then get all of the output from it?
When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
- 121
0
votes
0
answers
26
views
How are the self-attention layer weigths updated?
I am trying to figure out how the updating of the weights in the self attention layer works.
I think I have some basic understanding of how the self-attention mechanism works, however it is really ...
- 121
0
votes
1
answer
269
views
Literature to understand the components of Temporal Fusion Transformer
I'm currently reading the paper Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: https://arxiv.org/abs/1912.09363v3
However, I had to stop at page 7/8, which ...
- 1,508
0
votes
0
answers
31
views
Issue related to Transformer decoder druing Inference using all previous output tokens until each decoder time step
I've been trying to understand the shapes used during decoder (both self-attention and enc-dec-attention blocks) and understand there is a difference in the way decoder runs during training versus ...
- 309
0
votes
0
answers
8
views
Why do faster (eg sparse) versions of Transformers focus on the query-key product?
A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism:
$ softmax(\frac{Q K^T}{\sqrt{d}}) V $,
As I understand it, the runtime, assuming $\{Q, K,...
- 219
1
vote
0
answers
15
views
In attention models, why is dropout used for positional encodings?
In the "Attention is all you need" paper, they write:
we apply dropout to the sums of the embeddings and the positional encodings
I can understand why you might use dropout on the ...
- 613
3
votes
1
answer
47
views
Operation modes in neural turing machine (Graves, 2014)
I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out.
More ...
- 75
0
votes
1
answer
36
views
What is purpose of multihead in transformers
Why in the attention mechanism do they apply multihead?
It's been said that it would lift the amplitude of vector dot production from word that at the moment is analysed, which can be correct, but ...
- 141
0
votes
0
answers
79
views
Why transformer in deep learning is called transformer?
Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
- 173
0
votes
1
answer
127
views
Why the asymmetric design between (Q, K) and V in tranformer's attention?
In the Attention is all you need paper, the self-attention layer is defined as $\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V$.
I would like to know why a more ...
- 223
0
votes
0
answers
34
views
Recommendation for prediction of multivariate time series
I am looking for a recommendation (concrete package/framework/approach etc., preferably in Python/Keras or R as that is where I have experience) for predicting multivariate time series.
I do have ...
- 93
2
votes
1
answer
29
views
In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?
So attention and transformer models can be used for machine translation.
Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
- 365
0
votes
1
answer
173
views
How does tf.keras.MultiHeadAttention layer handle positional encoding?
In Attention Is All You Need paper, positional encodings are added to the input embeddings in order to consider the order of the sequence. How does tf.keras.MultiHeadAttention handle positional ...
1
vote
1
answer
125
views
Do we need Masking in test phase while we use transformer?
We use two types of masks when we train transformer models one is in the architecture of encoder to adjust for the length of the input sequence and another is the mask that is being used by the ...
- 121
0
votes
0
answers
27
views
Attention Scores in Seq2seq sorting example
This question is about the interpretation of the additive attention (Bahdanau) scores in a seq2seq problem concerning the sorting of an input numerical sequence.
I have a LSTM Encoder/Decoder model ...
- 1
0
votes
0
answers
16
views
Does swapping Query and Value in Attention make sense here?
I'm building a seq2seq transformer model that uses several convolutional-attention layers in both the encoder and the decoder. The CNN Attention layer takes an input and a 'state' and produces an ...
- 884
0
votes
0
answers
218
views
R Keras attention for 2-D timeseries in a classification task
I don't understand how a regular attention mechanism can be plugged in inside a neural architecture for a two-dimensional (a 2-D tensor) timeseries classification.
Let's say, my timeseries has length <...
- 2,479
0
votes
0
answers
43
views
Can the positional encoding in the transformer be called a 'stochastic' encoding?
I just want to make sure my intuition is correct in understanding how the positional encodings in "Attention Is All You Need" (https://arxiv.org/pdf/1706.03762.pdf) works...
Let the hidden ...
- 619
0
votes
0
answers
34
views
split in linear in multi-head attention
i just learned about transformer and until today i still got confused about somethings. after reading this article one and two
there are 2 things i dont understand
if in the case is 8 head attention, ...
- 1
0
votes
0
answers
221
views
The last question about query-key-value matrices in Transformer
There are a lot of questions regarding query-key-value concept in Transformers, however, the concept and the details are still unclear (at least for me).
Here I will describe my questions in details, ...
- 265
0
votes
0
answers
118
views
Transformer FF-Network output and encoder-decoder attention
I am trying to follow through Jay Alammar's blogpost on transformers, but cannot quite grasp all the details.
Encoder-Attention:
So, as to my understanding, we are given eight different Query/Key/...
- 1
3
votes
0
answers
110
views
Why PyTorch MultiheadAttention is considered as activation function?
When I scroll all activation functions available on PyTorch package (here), I found that nn.MultiheadAttention is described there. Can you please explain why it's ...
- 131
2
votes
1
answer
305
views
Why is Bahdanau's attention sometimes called concat attention?
I am learning the intuition behind the attention mechanism from
https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
https://lilianweng.github....
- 155
0
votes
0
answers
101
views
Can the encoder and decoder have different hidden dimensions in the Transformer architecture?
In the vanilla Transformer architecture, can we have an encoder with hidden size of 768, and a decoder with hidden size of 792?
1
vote
1
answer
295
views
How pre-trained weights in the BERT can help the fine tuning task?
I have been using the BERT architecture implemented by the Huggingface library for my sentence classification task. Although, I read the paper (and related papers) and the result of my experiments is ...
- 275
1
vote
1
answer
146
views
How does a transformer network's attention mechanism determine what to pay attention to?
The paper Attention is All You Need describes the transformer network which uses a "multi-head attention mechanism". The basic intuition behind this mechanism is that it weighs other tokens ...
4
votes
1
answer
339
views
When calculating self-attention for Transformer ML architectures, why do we need both a key and a query weight matrix?
I'm trying to understand the math behind Transformers, specifically self-attention. This link, and many others, gives the formula to compute the output vectors from the input embeddings as:
$$Q=XW_Q,\;...
- 43
6
votes
1
answer
3k
views
Deciding between Decoder-only or Encoder-only Transformers (BERT, GPT)
I just started learning about transformers and looked into the following 3 variants
The original one from Attention Is All You Need (Encoder & Decoder)
BERT (Encoder only)
GPT-2 (Decoder only)
...
- 169
1
vote
3
answers
788
views
transformer, linear layer after attention
My question regards this image:
It seems that after the multi head attention there is a linear layer as they mention also from here:
the linearity is given by the weights W^{o}. my quesion is: for ...
- 13
2
votes
0
answers
92
views
Attention dropout, where was it proposed/used first?
Attention dropout (dropout on the attention weights) is very common for the Transformer model. In the original Attention is all you need paper, dropout is mentioned, but not for the attention weights. ...
- 1,155
2
votes
1
answer
1k
views
What is masking in the attention if all you need paper?
I am a newbie to the NLP and specifically, the attention is all you need and I can understand the encoder part of the paper.
However, I am baffled about the decoder part. In the pic below and the ...
- 21
0
votes
1
answer
20
views
Attention is all you need: During run/test time, when output is not available, how is the decoder used? [closed]
In the famous paper "Attention is all you need" we see that in the Decoder we input the supposedly 'Output' sentence embeddings. During inference/test time, this output would not be ...
- 107
2
votes
0
answers
272
views
In the attention mechanism, why are there separate weight matrices for the queries and keys?
To perform self attention over a collection of $n$ vectors stacked up into a matrix $X \in \mathbb{R}^{n \times d}$, we first obtain query, key, and value representations of these vectors via three ...
- 3,205
0
votes
1
answer
50
views
Attention is all you need: During training, does decoder generates the whole sentence in 1 shot and not sequentially word by word?
In the famous paper "Attention is all you need", does the decoder generates the whole output sentence in one shot in parallel. In this case will the final softmax output the probabilities ...
- 107
0
votes
1
answer
57
views
Attention is all you need: from where does it get the encoder/decoder input embeddings?
In "Attention is all you need" paper, regarding encoder (and decoder) input embeddings: Do they use already pretrained such as off the shelf Word2vec or Glove embeddings ? or are they also ...
- 107
1
vote
0
answers
41
views
Any natural interpretation for "hopfield network is all you need"?
I am studying deep learning and came across to the paper claiming "so-called attention mechanism in deep learning is equivalent to hopfield network", in the paper called "Hopfield ...
- 139
2
votes
1
answer
391
views
Why use sinusoidal along embedding dimension in positional encoding in transformers?
Why do we need sinusoidal function along the embedding dimension in positional encoding in transformers? Shouldn't sinusoidal function along time dimension be enough?
This question is derived from ...
- 25
2
votes
1
answer
825
views
Does it make sense to use attention mechanism for seq-2-seq autoencoder for anomaly detection?
So I want to train LSTM sequence to sequence model, autoencoder, for anomaly detection. The idea is to train it on normal samples and when anomaly comes into model it will not be able to reconstruct ...
- 731