Questions tagged [attention]
Methods for aggregating a set feature vectors into a single feature vector relevant to a context.
106
questions
0
votes
0
answers
68
views
Difference between multi-head and single-head attention
Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
0
votes
0
answers
14
views
Alternatives to classical softmax for better aggregation
I am doing machine learning on phylogenetic trees. Species within the trees have several relations, such as living at the same time or not, and I would like to use a global attention mechanism which ...
- 43
0
votes
0
answers
21
views
Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"
I am fairly new to Transformers thus this schema confuses me a lot.
I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
1
vote
1
answer
275
views
In transformer architecture, Why does Masked Self Attention layer uses additive masks not multiplicative mask?
We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as
$$\alpha = Softmax(QK^T)$$
In the case of ...
- 11
0
votes
0
answers
16
views
How to classify each datapoint in a timeseries using deep learning
I am new to time series but for my project I wonder if there is something along those lines:
My aim is to classify each datapoint of a time-dependent time series. In practical terms, I have arrays of ...
- 1
4
votes
2
answers
116
views
Necessity of keys and queries in a decoder-only transformer model
when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$.
However, in the next step, we compute a dot product on the embedding ...
- 43
5
votes
2
answers
252
views
Attention Mechanisms in Tranformers, effect of using more parameters than Keys and Queries
In the attention mechanism, as described in the paper "Attention is all you need", normally the model learns weights $W_Q$ and $W_K$, i.e. the Queries and the Keys. The input of the ...
- 505
0
votes
0
answers
218
views
Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?
I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part.
In attention we use softmax ...
- 1,360
0
votes
1
answer
37
views
Effect of dK (key vector dimension) in transformers
Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers?
My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
0
votes
0
answers
20
views
How to encode sparse and variable length sequential data
I have 100k historical horse races. The data is sequential in time, so I am wishing to use online learning to train an LSTM (or sequential attention model or something similar...) such that the model ...
- 247
0
votes
0
answers
14
views
In the Transformer, how do I get the Keys and the Values from the output of the top encoder (which go into Encoder-Decoder Attention)? [duplicate]
I am trying to implement Attention is All You Need paper in PyTorch without looking at any code.
I'm struggling to understand how do I get the Keys and the Values from the output of the top encoder. ...
1
vote
1
answer
238
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
1
vote
1
answer
60
views
Intuitive difference between NN and attention for text prediction
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
1
vote
0
answers
26
views
Attention mechanism with different hidden states length?
i have two layer with different layer sizes (hidden states) how can i perform encoder decoder type of attention on these layers if the layer sizes are different? because i will do dot product, how?
...
- 11
0
votes
0
answers
31
views
Segmentation network for melanoma can't generalize well [duplicate]
I'm trying to replicate a model from a paper, and cannot make it generalize well. I'm using ISIC 2018 dataset for melanoma semantic segmentation. The masks are binary indicating in white what is ...
0
votes
1
answer
45
views
Do attention models change the sequences attended to for every novel sentence? Or do they have a general idea of where to look?
I've learned about many different types of attention models. But the one thing that I never understood was how they knew where to focus on/attend to in the sentence.
What I'm wondering is if they can ...
- 3
1
vote
0
answers
42
views
Linear positional encoding for Transformers with fixed sequence length
I am trying to use Transformers for a Time-series task (with fixed sequence length).
Since it's sequence length is always the same, can I use a linear positional encoding? I read some articles about ...
- 113
1
vote
1
answer
1k
views
Training Transformers: self attention weights vs embedding layer
I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
- 113
0
votes
0
answers
23
views
Is the attention values of different encoder layers comparable?
I’m new to this community. I studied a little theory on the attention mechanism, and now I would like to jump into practice. At this moment, I’m looking at the attention extracted from a BertModel. I’...
- 1
0
votes
0
answers
95
views
Using masks in pure encoder transformer networks
Why do we use masking in a (text / image) transformer architecture that consists of several stacked encoder modules?
Let's say I want to train an autoregressive model to predict the next token in a ...
- 645
1
vote
1
answer
572
views
How are the weight matrices in self-attention learned?
Learning the weights of logistic regression using gradient descent is quite intuitive. The input $x$ is multiplied with the weight $w$ to produce $y$, and we know the true target $\hat{y}$. Therefore, ...
- 500
0
votes
0
answers
5
views
What could be reason that the deep model is consuming approximately same test time with the same model with 1 more layer?
I was analyzing about test time needed by different deep learning algorithms. I use a CNNLSTM architecture (4 convolutional layers followed by 2 lstm layers for the classification task).
Let's say ...
- 171
0
votes
0
answers
64
views
Mask in transformers and its relation with sparse attention
Can anyone please explain in a clear way what is the usage of mask in attention for sparse attention? I just can not get how masking tokens (I do not mean here pad tokens) can make attention more fast ...
- 121
1
vote
0
answers
120
views
Deep learning model for set of sequences or multiple sequences?
My problem is this: I have a set of sequences as input for the model, each sequence contains vectors of length n (constant length for all sequences). Different sequences can have different lengths.
...
- 11
0
votes
0
answers
886
views
Proper masking in MultiHeadAttention layer in Keras
I am new to Transformers and I am trying to create a very simple model (not NLP area) for processing data of variable length (not sequence data because for my problem order in data does not matter). ...
- 11
1
vote
1
answer
457
views
Is Transformer multi-headed attention a form of ensemble?
In Transformer, is it correct to say that multi-headed attention is an ensemble learning and each head is learning a different capability by below?
independent random weight initialization for each ...
- 1,344
2
votes
0
answers
286
views
Is it possible to use attention in non sequential data in neural networks?
I'm still trying to understand the attention mechanism. It is not clear to me what query, key, and value mean yet, for example.
However, my main issue is regarding how to apply attention in my use ...
- 678
1
vote
1
answer
80
views
Does the attention mechanism (in CNNs) bring additional parameters/weights to learn to the network?
The idea of the attention mechanism is based on using some weighted sum of the output of some layers in deep networks. I see the process in forward propagation, and it seems that the attention ...
- 171
0
votes
1
answer
524
views
How to properly mask MultiHeadAttention for sliding window time series data
I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent ...
- 165
1
vote
1
answer
2k
views
How to obtain Key, Value and Query in Attention and Multi-Head-Attention
I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and ...
- 131
0
votes
2
answers
450
views
Do value and key of additive attention need to have the same dimension?
For the documented tensorflow-keras implementation of additive attention, it is stated that the input tensors are:
query: Query Tensor of shape ...
- 101
4
votes
2
answers
466
views
About basic understanding of attention mechanism and model weights
In the image domain, for a given image, suppose that we want to understand if there is a bird on that image (0 and 1 label = no bird and bird), attention mechanism helps to pay more attention to the ...
- 171
1
vote
1
answer
2k
views
Trainable weights in attention mechanism
I am wondering what are the trainable weights inside an attention-powered transformer. I figure the feed-forward layer contain trainable weights and the token embeddings, but what other parts contain ...
15
votes
1
answer
8k
views
Why are residual connections needed in transformer architectures?
Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training.
This problem is ...
- 287
2
votes
2
answers
83
views
Why the sequence of matrix transformations in The Annotated Transformer?
Forgive me the long introduction, but I want to be very specific with my question.
The Annotated Transformer implements the Transformer architecture from "Attention Is All You Need". The <...
- 21
2
votes
1
answer
81
views
Hard attention derivations
I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
- 47
1
vote
0
answers
97
views
How can one feed all of the input to an RNN, and then get all of the output from it?
When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
- 121
0
votes
0
answers
109
views
How are the self-attention layer weigths updated?
I am trying to figure out how the updating of the weights in the self attention layer works.
I think I have some basic understanding of how the self-attention mechanism works, however it is really ...
- 203
1
vote
1
answer
895
views
Literature to understand the components of Temporal Fusion Transformer
I'm currently reading the paper Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: https://arxiv.org/abs/1912.09363v3
However, I had to stop at page 7/8, which ...
- 1,606
2
votes
0
answers
34
views
Why do faster (eg sparse) versions of Transformers focus on the query-key product?
A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism:
$ softmax(\frac{Q K^T}{\sqrt{d}}) V $,
As I understand it, the runtime, assuming $\{Q, K,...
- 290
1
vote
0
answers
528
views
In attention models, why is dropout used for positional encodings?
In the "Attention is all you need" paper, they write:
we apply dropout to the sums of the embeddings and the positional encodings
I can understand why you might use dropout on the ...
- 613
3
votes
1
answer
62
views
Operation modes in neural turing machine (Graves, 2014)
I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out.
More ...
- 77
1
vote
1
answer
92
views
What is purpose of multihead in transformers
Why in the attention mechanism do they apply multihead?
It's been said that it would lift the amplitude of vector dot production from word that at the moment is analysed, which can be correct, but ...
- 169
6
votes
1
answer
2k
views
Why transformer in deep learning is called transformer?
Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
- 283
0
votes
1
answer
1k
views
Why the asymmetric design between (Q, K) and V in tranformer's attention?
In the Attention is all you need paper, the self-attention layer is defined as $\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V$.
I would like to know why a more ...
- 253
2
votes
1
answer
53
views
In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?
So attention and transformer models can be used for machine translation.
Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
- 395
0
votes
1
answer
401
views
How does tf.keras.MultiHeadAttention layer handle positional encoding?
In Attention Is All You Need paper, positional encodings are added to the input embeddings in order to consider the order of the sequence. How does tf.keras.MultiHeadAttention handle positional ...
1
vote
1
answer
1k
views
Do we need Masking in test phase while we use transformer?
We use two types of masks when we train transformer models one is in the architecture of encoder to adjust for the length of the input sequence and another is the mask that is being used by the ...
- 131
3
votes
0
answers
157
views
Why PyTorch MultiheadAttention is considered as activation function?
When I scroll all activation functions available on PyTorch package (here), I found that nn.MultiheadAttention is described there. Can you please explain why it's ...
- 131
3
votes
1
answer
925
views
Why is Bahdanau's attention sometimes called concat attention?
I am learning the intuition behind the attention mechanism from
https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
https://lilianweng.github....
- 165