Questions tagged [attention]
Methods for aggregating a set feature vectors into a single feature vector relevant to a context.
106 questions
0
votes
0
answers
25
views
Is attention a kind of K nearest neighbors regressor?
I came up with a strange KNN algorithm that I think is equivalent to self-attention:
Suppose we have a typical (features, label)-style dataset $\mathscr D = \{(x_1, Vx_1), (x_2, Vx_2),\dots, (x_N, ...
- 342
1
vote
1
answer
40
views
Attention, FFN Add&Norm Order
The actual structure Add to X Attention, then FFN.
I my structure on the right image, only the tiny $ \delta e$ generated by the FFN is added to ...
- 23
0
votes
0
answers
8
views
Measuring statistical difference across two distributions of attention scores
For a study I'm conducting, I have two distributions of attention scores generated via a trained transformer model. I'm plotting the two distributions and I would like to run a statistical test to ...
- 659
0
votes
0
answers
35
views
Why learn an embedding before self attention when training transformers?
I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
0
votes
0
answers
12
views
Combining both regression training and classification Evaluation for a single task
I was given a task which involved video summarization using AI. My initial thought was to use a basic classification model based on image features. However, after reading papers and conducting some ...
0
votes
1
answer
40
views
Does a model learn the same attention scores when retrained?
As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers?
It feels ...
- 11
0
votes
1
answer
108
views
Dimension Mismatch in Transformer Decoder: What Are the Input and Output Dimensions?
My understanding is that in the decoder, the output of the masked self-attention mechanism is expected to have dimensions (o_len,d_model), where o_len is the current output length.
However, an issue ...
- 1
0
votes
0
answers
97
views
Why is the maximum path length for convolutional layer $O(n/k)$ in attention is all you need paper?
In the table-1 third row it is being mentioned. Why is it $O(n/k)$? Take for example 1d convolution of 2 over 9 tokens with stride $1$. It won't be $n/k$ or $9/2=4.5$ rather it would be roughly $n-1$ ...
0
votes
1
answer
194
views
Understanding the function of attention layers in a convolutional neural network (U-Net in a diffusion model)
I am trying to understand the neural network architecture used by Ho et al. in "Denoising Diffusion Probabilistic Models" (paper, source code). They include self-attention layers in the ...
2
votes
0
answers
29
views
Contrast Self attention and Attention in machine learning for non technical person?
How would you contrast Self attention and Attention in machine learning for a non technical person?
I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
- 1,021
3
votes
1
answer
457
views
Difference between multi-head and single-head attention
Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
- 51
0
votes
0
answers
64
views
Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"
I am fairly new to Transformers thus this schema confuses me a lot.
I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
2
votes
1
answer
1k
views
In transformer architecture, Why does Masked Self Attention layer uses additive masks not multiplicative mask?
We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as
$$\alpha = Softmax(QK^T)$$
In the case of ...
- 31
4
votes
2
answers
563
views
Necessity of keys and queries in a decoder-only transformer model
when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$.
However, in the next step, we compute a dot product on the embedding ...
- 43
6
votes
2
answers
742
views
Attention Mechanisms in Tranformers, effect of using more parameters than Keys and Queries
In the attention mechanism, as described in the paper "Attention is all you need", normally the model learns weights $W_Q$ and $W_K$, i.e. the Queries and the Keys. The input of the ...
- 535
1
vote
1
answer
726
views
Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?
I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part.
In attention we use softmax ...
- 1,382
2
votes
1
answer
414
views
Effect of dK (key vector dimension) in transformers
Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers?
My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
1
vote
1
answer
1k
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
1
vote
1
answer
94
views
Intuitive difference between NN and attention for text prediction
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
1
vote
0
answers
51
views
Attention mechanism with different hidden states length?
i have two layer with different layer sizes (hidden states) how can i perform encoder decoder type of attention on these layers if the layer sizes are different? because i will do dot product, how?
...
- 11
1
vote
1
answer
71
views
Do attention models change the sequences attended to for every novel sentence? Or do they have a general idea of where to look?
I've learned about many different types of attention models. But the one thing that I never understood was how they knew where to focus on/attend to in the sentence.
What I'm wondering is if they can ...
- 13
1
vote
0
answers
75
views
Linear positional encoding for Transformers with fixed sequence length
I am trying to use Transformers for a Time-series task (with fixed sequence length).
Since it's sequence length is always the same, can I use a linear positional encoding? I read some articles about ...
- 113
4
votes
1
answer
4k
views
Training Transformers: self attention weights vs embedding layer
I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
- 143
2
votes
1
answer
1k
views
How are the weight matrices in self-attention learned?
Learning the weights of logistic regression using gradient descent is quite intuitive. The input $x$ is multiplied with the weight $w$ to produce $y$, and we know the true target $\hat{y}$. Therefore, ...
- 644
2
votes
0
answers
158
views
Deep learning model for set of sequences or multiple sequences?
My problem is this: I have a set of sequences as input for the model, each sequence contains vectors of length n (constant length for all sequences). Different sequences can have different lengths.
...
- 21
1
vote
1
answer
709
views
Is Transformer multi-headed attention a form of ensemble?
In Transformer, is it correct to say that multi-headed attention is an ensemble learning and each head is learning a different capability by below?
independent random weight initialization for each ...
- 1,568
2
votes
0
answers
557
views
Is it possible to use attention in non sequential data in neural networks?
I'm still trying to understand the attention mechanism. It is not clear to me what query, key, and value mean yet, for example.
However, my main issue is regarding how to apply attention in my use ...
- 1,008
1
vote
1
answer
113
views
Does the attention mechanism (in CNNs) bring additional parameters/weights to learn to the network?
The idea of the attention mechanism is based on using some weighted sum of the output of some layers in deep networks. I see the process in forward propagation, and it seems that the attention ...
- 243
1
vote
1
answer
644
views
How to properly mask MultiHeadAttention for sliding window time series data
I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent ...
- 185
1
vote
1
answer
4k
views
How to obtain Key, Value and Query in Attention and Multi-Head-Attention
I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and ...
- 131
1
vote
2
answers
861
views
Do value and key of additive attention need to have the same dimension?
For the documented tensorflow-keras implementation of additive attention, it is stated that the input tensors are:
query: Query Tensor of shape ...
- 141
5
votes
2
answers
765
views
About basic understanding of attention mechanism and model weights
In the image domain, for a given image, suppose that we want to understand if there is a bird on that image (0 and 1 label = no bird and bird), attention mechanism helps to pay more attention to the ...
- 243
1
vote
1
answer
3k
views
Trainable weights in attention mechanism
I am wondering what are the trainable weights inside an attention-powered transformer. I figure the feed-forward layer contain trainable weights and the token embeddings, but what other parts contain ...
19
votes
1
answer
17k
views
Why are residual connections needed in transformer architectures?
Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training.
This problem is ...
- 337
2
votes
2
answers
135
views
Why the sequence of matrix transformations in The Annotated Transformer?
Forgive me the long introduction, but I want to be very specific with my question.
The Annotated Transformer implements the Transformer architecture from "Attention Is All You Need". The <...
- 21
2
votes
1
answer
98
views
Hard attention derivations
I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
- 47
1
vote
0
answers
135
views
How can one feed all of the input to an RNN, and then get all of the output from it?
When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
- 139
0
votes
0
answers
228
views
How are the self-attention layer weigths updated?
I am trying to figure out how the updating of the weights in the self attention layer works.
I think I have some basic understanding of how the self-attention mechanism works, however it is really ...
- 233
1
vote
1
answer
1k
views
Literature to understand the components of Temporal Fusion Transformer
I'm currently reading the paper Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: https://arxiv.org/abs/1912.09363v3
However, I had to stop at page 7/8, which ...
- 1,658
2
votes
0
answers
37
views
Why do faster (eg sparse) versions of Transformers focus on the query-key product?
A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism:
$ softmax(\frac{Q K^T}{\sqrt{d}}) V $,
As I understand it, the runtime, assuming $\{Q, K,...
- 350
2
votes
0
answers
841
views
In attention models, why is dropout used for positional encodings?
In the "Attention is all you need" paper, they write:
we apply dropout to the sums of the embeddings and the positional encodings
I can understand why you might use dropout on the ...
- 663
3
votes
1
answer
65
views
Operation modes in neural turing machine (Graves, 2014)
I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out.
More ...
- 77
1
vote
1
answer
107
views
What is purpose of multihead in transformers
Why in the attention mechanism do they apply multihead?
It's been said that it would lift the amplitude of vector dot production from word that at the moment is analysed, which can be correct, but ...
- 169
9
votes
2
answers
4k
views
Why transformer in deep learning is called transformer?
Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
- 343
0
votes
1
answer
2k
views
Why the asymmetric design between (Q, K) and V in tranformer's attention?
In the Attention is all you need paper, the self-attention layer is defined as $\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V$.
I would like to know why a more ...
- 253
2
votes
1
answer
59
views
In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?
So attention and transformer models can be used for machine translation.
Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
- 395
0
votes
1
answer
504
views
How does tf.keras.MultiHeadAttention layer handle positional encoding?
In Attention Is All You Need paper, positional encodings are added to the input embeddings in order to consider the order of the sequence. How does tf.keras.MultiHeadAttention handle positional ...
1
vote
1
answer
2k
views
Do we need Masking in test phase while we use transformer?
We use two types of masks when we train transformer models one is in the architecture of encoder to adjust for the length of the input sequence and another is the mask that is being used by the ...
- 131
3
votes
0
answers
195
views
Why PyTorch MultiheadAttention is considered as activation function?
When I scroll all activation functions available on PyTorch package (here), I found that nn.MultiheadAttention is described there. Can you please explain why it's ...
- 131
3
votes
1
answer
1k
views
Why is Bahdanau's attention sometimes called concat attention?
I am learning the intuition behind the attention mechanism from
https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
https://lilianweng.github....
- 165