# Questions tagged [attention]

Methods for aggregating a set feature vectors into a single feature vector relevant to a context.

105 questions
Filter by
Sorted by
Tagged with
8 views

### Measuring statistical difference across two distributions of attention scores

For a study I'm conducting, I have two distributions of attention scores generated via a trained transformer model. I'm plotting the two distributions and I would like to run a statistical test to ...
• 637
9 views

### Why learn an embedding before self attention when training transformers?

I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
11 views

### Combining both regression training and classification Evaluation for a single task

I was given a task which involved video summarization using AI. My initial thought was to use a basic classification model based on image features. However, after reading papers and conducting some ...
27 views

### Does a model learn the same attention scores when retrained?

As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers? It feels ...
• 11
73 views

### Dimension Mismatch in Transformer Decoder: What Are the Input and Output Dimensions?

My understanding is that in the decoder, the output of the masked self-attention mechanism is expected to have dimensions (o_len,d_model), where o_len is the current output length. However, an issue ...
67 views

### Why is the maximum path length for convolutional layer $O(n/k)$ in attention is all you need paper?

In the table-1 third row it is being mentioned. Why is it $O(n/k)$? Take for example 1d convolution of 2 over 9 tokens with stride $1$. It won't be $n/k$ or $9/2=4.5$ rather it would be roughly $n-1$ ...
133 views

### Understanding the function of attention layers in a convolutional neural network (U-Net in a diffusion model)

I am trying to understand the neural network architecture used by Ho et al. in "Denoising Diffusion Probabilistic Models" (paper, source code). They include self-attention layers in the ...
28 views

### Contrast Self attention and Attention in machine learning for non technical person?

How would you contrast Self attention and Attention in machine learning for a non technical person? I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
• 1,021
348 views

Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
81 views

### Alternatives to classical softmax for better aggregation

I am doing machine learning on phylogenetic trees. Species within the trees have several relations, such as living at the same time or not, and I would like to use a global attention mechanism which ...
• 43
46 views

### Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"

I am fairly new to Transformers thus this schema confuses me a lot. I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
1k views

We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as $$\alpha = Softmax(QK^T)$$ In the case of ...
460 views

### Necessity of keys and queries in a decoder-only transformer model

when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$. However, in the next step, we compute a dot product on the embedding ...
• 43
575 views

### Attention Mechanisms in Tranformers, effect of using more parameters than Keys and Queries

In the attention mechanism, as described in the paper "Attention is all you need", normally the model learns weights $W_Q$ and $W_K$, i.e. the Queries and the Keys. The input of the ...
• 535
1 vote
606 views

### Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?

I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part. In attention we use softmax ...
• 1,382
259 views

### Effect of dK (key vector dimension) in transformers

Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers? My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
1 vote
825 views

### How do I go from embeddings to queries, keys and values in the Transformer model?

I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
1 vote
85 views

### Intuitive difference between NN and attention for text prediction

If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things: Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
1 vote
39 views

### Attention mechanism with different hidden states length?

i have two layer with different layer sizes (hidden states) how can i perform encoder decoder type of attention on these layers if the layer sizes are different? because i will do dot product, how? ...
1 vote
58 views

### Do attention models change the sequences attended to for every novel sentence? Or do they have a general idea of where to look?

I've learned about many different types of attention models. But the one thing that I never understood was how they knew where to focus on/attend to in the sentence. What I'm wondering is if they can ...
• 13
1 vote
69 views

### Linear positional encoding for Transformers with fixed sequence length

I am trying to use Transformers for a Time-series task (with fixed sequence length). Since it's sequence length is always the same, can I use a linear positional encoding? I read some articles about ...
• 113
3k views

### Training Transformers: self attention weights vs embedding layer

I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
• 143
964 views

### How are the weight matrices in self-attention learned?

Learning the weights of logistic regression using gradient descent is quite intuitive. The input $x$ is multiplied with the weight $w$ to produce $y$, and we know the true target $\hat{y}$. Therefore, ...
1 vote
155 views

### Deep learning model for set of sequences or multiple sequences?

My problem is this: I have a set of sequences as input for the model, each sequence contains vectors of length n (constant length for all sequences). Different sequences can have different lengths. ...
• 11
1 vote
645 views

### Is Transformer multi-headed attention a form of ensemble?

In Transformer, is it correct to say that multi-headed attention is an ensemble learning and each head is learning a different capability by below? independent random weight initialization for each ...
• 1,548
493 views

### Is it possible to use attention in non sequential data in neural networks?

I'm still trying to understand the attention mechanism. It is not clear to me what query, key, and value mean yet, for example. However, my main issue is regarding how to apply attention in my use ...
• 1,018
1 vote
104 views

### Does the attention mechanism (in CNNs) bring additional parameters/weights to learn to the network?

The idea of the attention mechanism is based on using some weighted sum of the output of some layers in deep networks. I see the process in forward propagation, and it seems that the attention ...
• 233
1 vote
627 views

### How to properly mask MultiHeadAttention for sliding window time series data

I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent ...
• 185
1 vote
3k views

### How to obtain Key, Value and Query in Attention and Multi-Head-Attention

I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and ...
• 131
1 vote
761 views

### Do value and key of additive attention need to have the same dimension?

For the documented tensorflow-keras implementation of additive attention, it is stated that the input tensors are: query: Query Tensor of shape ...
• 141
735 views

### About basic understanding of attention mechanism and model weights

In the image domain, for a given image, suppose that we want to understand if there is a bird on that image (0 and 1 label = no bird and bird), attention mechanism helps to pay more attention to the ...
• 233
1 vote
3k views

### Trainable weights in attention mechanism

I am wondering what are the trainable weights inside an attention-powered transformer. I figure the feed-forward layer contain trainable weights and the token embeddings, but what other parts contain ...
15k views

### Why are residual connections needed in transformer architectures?

Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training. This problem is ...
124 views

### Why the sequence of matrix transformations in The Annotated Transformer?

Forgive me the long introduction, but I want to be very specific with my question. The Annotated Transformer implements the Transformer architecture from "Attention Is All You Need". The <...
• 21
92 views

### Hard attention derivations

I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
• 47
1 vote
135 views

### How can one feed all of the input to an RNN, and then get all of the output from it?

When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
• 139
196 views

### How are the self-attention layer weigths updated?

I am trying to figure out how the updating of the weights in the self attention layer works. I think I have some basic understanding of how the self-attention mechanism works, however it is really ...
• 223
1 vote
1k views

### Literature to understand the components of Temporal Fusion Transformer

I'm currently reading the paper Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: https://arxiv.org/abs/1912.09363v3 However, I had to stop at page 7/8, which ...
• 1,658
37 views

58 views

### In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?

So attention and transformer models can be used for machine translation. Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
• 395
480 views

### How does tf.keras.MultiHeadAttention layer handle positional encoding?

In Attention Is All You Need paper, positional encodings are added to the input embeddings in order to consider the order of the sequence. How does tf.keras.MultiHeadAttention handle positional ...
1 vote
2k views

### Do we need Masking in test phase while we use transformer?

We use two types of masks when we train transformer models one is in the architecture of encoder to adjust for the length of the input sequence and another is the mask that is being used by the ...
• 131
190 views

### Why PyTorch MultiheadAttention is considered as activation function?

When I scroll all activation functions available on PyTorch package (here), I found that nn.MultiheadAttention is described there. Can you please explain why it's ...
• 131