Questions tagged [attention]

Methods for aggregating a set feature vectors into a single feature vector relevant to a context.

Filter by
Sorted by
Tagged with
0 votes
0 answers
68 views

Difference between multi-head and single-head attention

Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
tolgarecep's user avatar
0 votes
0 answers
14 views

Alternatives to classical softmax for better aggregation

I am doing machine learning on phylogenetic trees. Species within the trees have several relations, such as living at the same time or not, and I would like to use a global attention mechanism which ...
25laps's user avatar
  • 43
0 votes
0 answers
21 views

Confusing GPT architecture diagram from the paper "Locating and Editing Factual Associations in GPT"

I am fairly new to Transformers thus this schema confuses me a lot. I am familiar with "classic" "vanilla" encoder-decoder transformers. I have figured out that GPT-like models are ...
user3187119's user avatar
1 vote
1 answer
275 views

In transformer architecture, Why does Masked Self Attention layer uses additive masks not multiplicative mask?

We know that in transformer architecture, we first compute $Q,K$ and $V$ matrices from the token embeddings. Then we compute the attention weights $\alpha$ as $$\alpha = Softmax(QK^T)$$ In the case of ...
Arun prakash's user avatar
0 votes
0 answers
16 views

How to classify each datapoint in a timeseries using deep learning

I am new to time series but for my project I wonder if there is something along those lines: My aim is to classify each datapoint of a time-dependent time series. In practical terms, I have arrays of ...
Tria Ufo's user avatar
4 votes
2 answers
116 views

Necessity of keys and queries in a decoder-only transformer model

when we compute self-attention in a decoder model, we compute, for an embedding $x$, the tensors $Q = W_Q x, K = W_K x, V = W_V x$. However, in the next step, we compute a dot product on the embedding ...
25laps's user avatar
  • 43
5 votes
2 answers
252 views

Attention Mechanisms in Tranformers, effect of using more parameters than Keys and Queries

In the attention mechanism, as described in the paper "Attention is all you need", normally the model learns weights $W_Q$ and $W_K$, i.e. the Queries and the Keys. The input of the ...
kklaw's user avatar
  • 505
0 votes
0 answers
218 views

Why can't we replace the multi-head attention layer in transformers by a single head with sigmoid function?

I know that the reason we use multi-head attention in transformers instead of only single head is to attend to different parts of the input instead of just only one part. In attention we use softmax ...
floyd's user avatar
  • 1,360
0 votes
1 answer
37 views

Effect of dK (key vector dimension) in transformers

Intuitively speaking, what's the impact of changing the $d_k$ (and $d_{kv}$) for transformers? My understanding is that each attention head is effectively a lower-dimensional projection of a higher-...
Anonymous Scientist's user avatar
0 votes
0 answers
20 views

How to encode sparse and variable length sequential data

I have 100k historical horse races. The data is sequential in time, so I am wishing to use online learning to train an LSTM (or sequential attention model or something similar...) such that the model ...
Harry Stuart's user avatar
0 votes
0 answers
14 views

In the Transformer, how do I get the Keys and the Values from the output of the top encoder (which go into Encoder-Decoder Attention)? [duplicate]

I am trying to implement Attention is All You Need paper in PyTorch without looking at any code. I'm struggling to understand how do I get the Keys and the Values from the output of the top encoder. ...
Abysmal_query's user avatar
1 vote
1 answer
238 views

How do I go from embeddings to queries, keys and values in the Transformer model?

I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
Abysmal_query's user avatar
1 vote
1 answer
60 views

Intuitive difference between NN and attention for text prediction

If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things: Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
tunafriedrice's user avatar
1 vote
0 answers
26 views

Attention mechanism with different hidden states length?

i have two layer with different layer sizes (hidden states) how can i perform encoder decoder type of attention on these layers if the layer sizes are different? because i will do dot product, how? ...
user377324's user avatar
0 votes
0 answers
31 views

Segmentation network for melanoma can't generalize well [duplicate]

I'm trying to replicate a model from a paper, and cannot make it generalize well. I'm using ISIC 2018 dataset for melanoma semantic segmentation. The masks are binary indicating in white what is ...
Rodrigo Kapobel's user avatar
0 votes
1 answer
45 views

Do attention models change the sequences attended to for every novel sentence? Or do they have a general idea of where to look?

I've learned about many different types of attention models. But the one thing that I never understood was how they knew where to focus on/attend to in the sentence. What I'm wondering is if they can ...
Harvey's user avatar
  • 3
1 vote
0 answers
42 views

Linear positional encoding for Transformers with fixed sequence length

I am trying to use Transformers for a Time-series task (with fixed sequence length). Since it's sequence length is always the same, can I use a linear positional encoding? I read some articles about ...
Wenuka's user avatar
  • 113
1 vote
1 answer
1k views

Training Transformers: self attention weights vs embedding layer

I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
Sami Wood's user avatar
  • 113
0 votes
0 answers
23 views

Is the attention values of different encoder layers comparable?

I’m new to this community. I studied a little theory on the attention mechanism, and now I would like to jump into practice. At this moment, I’m looking at the attention extracted from a BertModel. I’...
Frap's user avatar
  • 1
0 votes
0 answers
95 views

Using masks in pure encoder transformer networks

Why do we use masking in a (text / image) transformer architecture that consists of several stacked encoder modules? Let's say I want to train an autoregressive model to predict the next token in a ...
Gilfoyle's user avatar
  • 645
1 vote
1 answer
572 views

How are the weight matrices in self-attention learned?

Learning the weights of logistic regression using gradient descent is quite intuitive. The input $x$ is multiplied with the weight $w$ to produce $y$, and we know the true target $\hat{y}$. Therefore, ...
desert_ranger's user avatar
0 votes
0 answers
5 views

What could be reason that the deep model is consuming approximately same test time with the same model with 1 more layer?

I was analyzing about test time needed by different deep learning algorithms. I use a CNNLSTM architecture (4 convolutional layers followed by 2 lstm layers for the classification task). Let's say ...
Mas A's user avatar
  • 171
0 votes
0 answers
64 views

Mask in transformers and its relation with sparse attention

Can anyone please explain in a clear way what is the usage of mask in attention for sparse attention? I just can not get how masking tokens (I do not mean here pad tokens) can make attention more fast ...
Ариж Аль Адел's user avatar
1 vote
0 answers
120 views

Deep learning model for set of sequences or multiple sequences?

My problem is this: I have a set of sequences as input for the model, each sequence contains vectors of length n (constant length for all sequences). Different sequences can have different lengths. ...
antivujo's user avatar
0 votes
0 answers
886 views

Proper masking in MultiHeadAttention layer in Keras

I am new to Transformers and I am trying to create a very simple model (not NLP area) for processing data of variable length (not sequence data because for my problem order in data does not matter). ...
antivujo's user avatar
1 vote
1 answer
457 views

Is Transformer multi-headed attention a form of ensemble?

In Transformer, is it correct to say that multi-headed attention is an ensemble learning and each head is learning a different capability by below? independent random weight initialization for each ...
mon's user avatar
  • 1,344
2 votes
0 answers
286 views

Is it possible to use attention in non sequential data in neural networks?

I'm still trying to understand the attention mechanism. It is not clear to me what query, key, and value mean yet, for example. However, my main issue is regarding how to apply attention in my use ...
Zaratruta's user avatar
  • 678
1 vote
1 answer
80 views

Does the attention mechanism (in CNNs) bring additional parameters/weights to learn to the network?

The idea of the attention mechanism is based on using some weighted sum of the output of some layers in deep networks. I see the process in forward propagation, and it seems that the attention ...
Mas A's user avatar
  • 171
0 votes
1 answer
524 views

How to properly mask MultiHeadAttention for sliding window time series data

I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent ...
JakeTheSnake's user avatar
1 vote
1 answer
2k views

How to obtain Key, Value and Query in Attention and Multi-Head-Attention

I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and ...
Josef's user avatar
  • 131
0 votes
2 answers
450 views

Do value and key of additive attention need to have the same dimension?

For the documented tensorflow-keras implementation of additive attention, it is stated that the input tensors are: query: Query Tensor of shape ...
arivero's user avatar
  • 101
4 votes
2 answers
466 views

About basic understanding of attention mechanism and model weights

In the image domain, for a given image, suppose that we want to understand if there is a bird on that image (0 and 1 label = no bird and bird), attention mechanism helps to pay more attention to the ...
Mas A's user avatar
  • 171
1 vote
1 answer
2k views

Trainable weights in attention mechanism

I am wondering what are the trainable weights inside an attention-powered transformer. I figure the feed-forward layer contain trainable weights and the token embeddings, but what other parts contain ...
Glam Casvaluir's user avatar
15 votes
1 answer
8k views

Why are residual connections needed in transformer architectures?

Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training. This problem is ...
Ramiro Hum-Sah's user avatar
2 votes
2 answers
83 views

Why the sequence of matrix transformations in The Annotated Transformer?

Forgive me the long introduction, but I want to be very specific with my question. The Annotated Transformer implements the Transformer architecture from "Attention Is All You Need". The <...
Lukasz's user avatar
  • 21
2 votes
1 answer
81 views

Hard attention derivations

I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
Grumpy C's user avatar
1 vote
0 answers
97 views

How can one feed all of the input to an RNN, and then get all of the output from it?

When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
HappyFace's user avatar
  • 121
0 votes
0 answers
109 views

How are the self-attention layer weigths updated?

I am trying to figure out how the updating of the weights in the self attention layer works. I think I have some basic understanding of how the self-attention mechanism works, however it is really ...
ClonedOne's user avatar
  • 203
1 vote
1 answer
895 views

Literature to understand the components of Temporal Fusion Transformer

I'm currently reading the paper Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: https://arxiv.org/abs/1912.09363v3 However, I had to stop at page 7/8, which ...
Patrick Bormann's user avatar
2 votes
0 answers
34 views

Why do faster (eg sparse) versions of Transformers focus on the query-key product?

A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism: $ softmax(\frac{Q K^T}{\sqrt{d}}) V $, As I understand it, the runtime, assuming $\{Q, K,...
Betterthan Kwora's user avatar
1 vote
0 answers
528 views

In attention models, why is dropout used for positional encodings?

In the "Attention is all you need" paper, they write: we apply dropout to the sums of the embeddings and the positional encodings I can understand why you might use dropout on the ...
user1834069's user avatar
3 votes
1 answer
62 views

Operation modes in neural turing machine (Graves, 2014)

I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out. More ...
Alf's user avatar
  • 77
1 vote
1 answer
92 views

What is purpose of multihead in transformers

Why in the attention mechanism do they apply multihead? It's been said that it would lift the amplitude of vector dot production from word that at the moment is analysed, which can be correct, but ...
Farhang Amaji's user avatar
6 votes
1 answer
2k views

Why transformer in deep learning is called transformer?

Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
Johnny Tam's user avatar
0 votes
1 answer
1k views

Why the asymmetric design between (Q, K) and V in tranformer's attention?

In the Attention is all you need paper, the self-attention layer is defined as $\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V$. I would like to know why a more ...
Joannes Vermorel's user avatar
2 votes
1 answer
53 views

In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?

So attention and transformer models can be used for machine translation. Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
Kadaj13's user avatar
  • 395
0 votes
1 answer
401 views

How does tf.keras.MultiHeadAttention layer handle positional encoding?

In Attention Is All You Need paper, positional encodings are added to the input embeddings in order to consider the order of the sequence. How does tf.keras.MultiHeadAttention handle positional ...
DiMorten - Jorge Chamorro's user avatar
1 vote
1 answer
1k views

Do we need Masking in test phase while we use transformer?

We use two types of masks when we train transformer models one is in the architecture of encoder to adjust for the length of the input sequence and another is the mask that is being used by the ...
Varun Singh's user avatar
3 votes
0 answers
157 views

Why PyTorch MultiheadAttention is considered as activation function?

When I scroll all activation functions available on PyTorch package (here), I found that nn.MultiheadAttention is described there. Can you please explain why it's ...
demo's user avatar
  • 131
3 votes
1 answer
925 views

Why is Bahdanau's attention sometimes called concat attention?

I am learning the intuition behind the attention mechanism from https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/ https://lilianweng.github....
bluesmonk's user avatar
  • 165