Skip to main content

Questions tagged [attention]

Methods for aggregating a set feature vectors into a single feature vector relevant to a context.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Measuring statistical difference across two distributions of attention scores

For a study I'm conducting, I have two distributions of attention scores generated via a trained transformer model. I'm plotting the two distributions and I would like to run a statistical test to ...
James Arten's user avatar
0 votes
0 answers
9 views

Why learn an embedding before self attention when training transformers?

I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
Nicolas Johnson's user avatar
0 votes
0 answers
11 views

Combining both regression training and classification Evaluation for a single task

I was given a task which involved video summarization using AI. My initial thought was to use a basic classification model based on image features. However, after reading papers and conducting some ...
moha tech's user avatar
0 votes
1 answer
27 views

Does a model learn the same attention scores when retrained?

As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers? It feels ...
rick's user avatar
  • 11
0 votes
1 answer
73 views

Dimension Mismatch in Transformer Decoder: What Are the Input and Output Dimensions?

My understanding is that in the decoder, the output of the masked self-attention mechanism is expected to have dimensions (o_len,d_model), where o_len is the current output length. However, an issue ...
ENing's user avatar
  • 1
0 votes
0 answers
67 views

Why is the maximum path length for convolutional layer $O(n/k)$ in attention is all you need paper?

In the table-1 third row it is being mentioned. Why is it $O(n/k)$? Take for example 1d convolution of 2 over 9 tokens with stride $1$. It won't be $n/k$ or $9/2=4.5$ rather it would be roughly $n-1$ ...
user404316's user avatar
0 votes
1 answer
133 views

Understanding the function of attention layers in a convolutional neural network (U-Net in a diffusion model)

I am trying to understand the neural network architecture used by Ho et al. in "Denoising Diffusion Probabilistic Models" (paper, source code). They include self-attention layers in the ...
Rational Function's user avatar
2 votes
0 answers
28 views

Contrast Self attention and Attention in machine learning for non technical person?

How would you contrast Self attention and Attention in machine learning for a non technical person? I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
Katsu's user avatar
  • 1,021
0 votes
1 answer
348 views

Difference between multi-head and single-head attention

Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
tolgarecep's user avatar
0 votes
0 answers
81 views

Alternatives to classical softmax for better aggregation

I am doing machine learning on phylogenetic trees. Species within the trees have several relations, such as living at the same time or not, and I would like to use a global attention mechanism which ...
25laps's user avatar
  • 43
0 votes
0 answers
46 views

Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"

I am fairly new to Transformers thus this schema confuses me a lot. I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
user3187119's user avatar
2 votes
1 answer
1k views

In transformer architecture, Why does Masked Self Attention layer uses additive masks not multiplicative mask?

We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as $$\alpha = Softmax(QK^T)$$ In the case of ...
Arun prakash's user avatar
4 votes
2 answers
460 views

Necessity of keys and queries in a decoder-only transformer model

when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$. However, in the next step, we compute a dot product on the embedding ...
25laps's user avatar
  • 43
6 votes
2 answers
575 views

Attention Mechanisms in Tranformers, effect of using more parameters than Keys and Queries

In the attention mechanism, as described in the paper "Attention is all you need", normally the model learns weights $W_Q$ and $W_K$, i.e. the Queries and the Keys. The input of the ...
kklaw's user avatar
  • 535
1 vote
1 answer
606 views

Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?

I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part. In attention we use softmax ...
floyd's user avatar
  • 1,382
0 votes
1 answer
259 views

Effect of dK (key vector dimension) in transformers

Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers? My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
Anonymous Scientist's user avatar
1 vote
1 answer
825 views

How do I go from embeddings to queries, keys and values in the Transformer model?

I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
Abysmal_query's user avatar
1 vote
1 answer
85 views

Intuitive difference between NN and attention for text prediction

If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things: Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
tunafriedrice's user avatar
1 vote
0 answers
39 views

Attention mechanism with different hidden states length?

i have two layer with different layer sizes (hidden states) how can i perform encoder decoder type of attention on these layers if the layer sizes are different? because i will do dot product, how? ...
user377324's user avatar
1 vote
1 answer
58 views

Do attention models change the sequences attended to for every novel sentence? Or do they have a general idea of where to look?

I've learned about many different types of attention models. But the one thing that I never understood was how they knew where to focus on/attend to in the sentence. What I'm wondering is if they can ...
Harvey's user avatar
  • 13
1 vote
0 answers
69 views

Linear positional encoding for Transformers with fixed sequence length

I am trying to use Transformers for a Time-series task (with fixed sequence length). Since it's sequence length is always the same, can I use a linear positional encoding? I read some articles about ...
Wenuka's user avatar
  • 113
4 votes
1 answer
3k views

Training Transformers: self attention weights vs embedding layer

I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
Sami Wood's user avatar
  • 143
2 votes
1 answer
964 views

How are the weight matrices in self-attention learned?

Learning the weights of logistic regression using gradient descent is quite intuitive. The input $x$ is multiplied with the weight $w$ to produce $y$, and we know the true target $\hat{y}$. Therefore, ...
desert_ranger's user avatar
1 vote
0 answers
155 views

Deep learning model for set of sequences or multiple sequences?

My problem is this: I have a set of sequences as input for the model, each sequence contains vectors of length n (constant length for all sequences). Different sequences can have different lengths. ...
antivujo's user avatar
1 vote
1 answer
645 views

Is Transformer multi-headed attention a form of ensemble?

In Transformer, is it correct to say that multi-headed attention is an ensemble learning and each head is learning a different capability by below? independent random weight initialization for each ...
mon's user avatar
  • 1,548
2 votes
0 answers
493 views

Is it possible to use attention in non sequential data in neural networks?

I'm still trying to understand the attention mechanism. It is not clear to me what query, key, and value mean yet, for example. However, my main issue is regarding how to apply attention in my use ...
Zaratruta's user avatar
  • 1,018
1 vote
1 answer
104 views

Does the attention mechanism (in CNNs) bring additional parameters/weights to learn to the network?

The idea of the attention mechanism is based on using some weighted sum of the output of some layers in deep networks. I see the process in forward propagation, and it seems that the attention ...
Mas A's user avatar
  • 233
1 vote
1 answer
627 views

How to properly mask MultiHeadAttention for sliding window time series data

I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent ...
JakeTheSnake's user avatar
1 vote
1 answer
3k views

How to obtain Key, Value and Query in Attention and Multi-Head-Attention

I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and ...
Josef's user avatar
  • 131
1 vote
2 answers
761 views

Do value and key of additive attention need to have the same dimension?

For the documented tensorflow-keras implementation of additive attention, it is stated that the input tensors are: query: Query Tensor of shape ...
arivero's user avatar
  • 141
5 votes
2 answers
735 views

About basic understanding of attention mechanism and model weights

In the image domain, for a given image, suppose that we want to understand if there is a bird on that image (0 and 1 label = no bird and bird), attention mechanism helps to pay more attention to the ...
Mas A's user avatar
  • 233
1 vote
1 answer
3k views

Trainable weights in attention mechanism

I am wondering what are the trainable weights inside an attention-powered transformer. I figure the feed-forward layer contain trainable weights and the token embeddings, but what other parts contain ...
Glam Casvaluir's user avatar
19 votes
1 answer
15k views

Why are residual connections needed in transformer architectures?

Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training. This problem is ...
Ramiro Hum-Sah's user avatar
2 votes
2 answers
124 views

Why the sequence of matrix transformations in The Annotated Transformer?

Forgive me the long introduction, but I want to be very specific with my question. The Annotated Transformer implements the Transformer architecture from "Attention Is All You Need". The <...
Lukasz's user avatar
  • 21
2 votes
1 answer
92 views

Hard attention derivations

I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
Grumpy C's user avatar
1 vote
0 answers
135 views

How can one feed all of the input to an RNN, and then get all of the output from it?

When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
HappyFace's user avatar
  • 139
0 votes
0 answers
196 views

How are the self-attention layer weigths updated?

I am trying to figure out how the updating of the weights in the self attention layer works. I think I have some basic understanding of how the self-attention mechanism works, however it is really ...
ClonedOne's user avatar
  • 223
1 vote
1 answer
1k views

Literature to understand the components of Temporal Fusion Transformer

I'm currently reading the paper Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: https://arxiv.org/abs/1912.09363v3 However, I had to stop at page 7/8, which ...
Patrick Bormann's user avatar
2 votes
0 answers
37 views

Why do faster (eg sparse) versions of Transformers focus on the query-key product?

A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism: $ softmax(\frac{Q K^T}{\sqrt{d}}) V $, As I understand it, the runtime, assuming $\{Q, K,...
Betterthan Kwora's user avatar
2 votes
0 answers
793 views

In attention models, why is dropout used for positional encodings?

In the "Attention is all you need" paper, they write: we apply dropout to the sums of the embeddings and the positional encodings I can understand why you might use dropout on the ...
user1834069's user avatar
3 votes
1 answer
65 views

Operation modes in neural turing machine (Graves, 2014)

I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out. More ...
Alf's user avatar
  • 77
1 vote
1 answer
103 views

What is purpose of multihead in transformers

Why in the attention mechanism do they apply multihead? It's been said that it would lift the amplitude of vector dot production from word that at the moment is analysed, which can be correct, but ...
Farhang Amaji's user avatar
7 votes
1 answer
3k views

Why transformer in deep learning is called transformer?

Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
Johnny Tam's user avatar
0 votes
1 answer
2k views

Why the asymmetric design between (Q, K) and V in tranformer's attention?

In the Attention is all you need paper, the self-attention layer is defined as $\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V$. I would like to know why a more ...
Joannes Vermorel's user avatar
2 votes
1 answer
58 views

In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?

So attention and transformer models can be used for machine translation. Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
Kadaj13's user avatar
  • 395
0 votes
1 answer
480 views

How does tf.keras.MultiHeadAttention layer handle positional encoding?

In Attention Is All You Need paper, positional encodings are added to the input embeddings in order to consider the order of the sequence. How does tf.keras.MultiHeadAttention handle positional ...
DiMorten - Jorge Chamorro's user avatar
1 vote
1 answer
2k views

Do we need Masking in test phase while we use transformer?

We use two types of masks when we train transformer models one is in the architecture of encoder to adjust for the length of the input sequence and another is the mask that is being used by the ...
Varun Singh's user avatar
3 votes
0 answers
190 views

Why PyTorch MultiheadAttention is considered as activation function?

When I scroll all activation functions available on PyTorch package (here), I found that nn.MultiheadAttention is described there. Can you please explain why it's ...
demo's user avatar
  • 131
3 votes
1 answer
1k views

Why is Bahdanau's attention sometimes called concat attention?

I am learning the intuition behind the attention mechanism from https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/ https://lilianweng.github....
bluesmonk's user avatar
  • 165
0 votes
0 answers
843 views

Can the encoder and decoder have different hidden dimensions in the Transformer architecture?

In the vanilla Transformer architecture, can we have an encoder with hidden size of 768, and a decoder with hidden size of 792?
Seongmin Park's user avatar