Questions tagged [attention]

Methods for aggregating a set feature vectors into a single feature vector relevant to a context.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

What does the weight matrix in dot-product attention capture about the training set?

Let $x, y \in \mathbb{R}^d$ be the input and output vectors in a seq2seq setup with one self-attention head. When we compute dot-product attention using queries and keys, given by, $q^Tk = x^TW_q^TW_k ...
user avatar
0 votes
0 answers
23 views

How does attention add expressive power to encoder-decoders?

I am learning about the attention mechanism for the first time, and the context in which it has been introduced (watching Lecture 8 of Stanford's CS224N) is that of language translation using seq2seq ...
user avatar
0 votes
1 answer
7 views

Does the attention mechanism (in CNNs) bring additional parameters/weights to learn to the network?

The idea of the attention mechanism is based on using some weighted sum of the output of some layers in deep networks. I see the process in forward propagation, and it seems that the attention ...
user avatar
  • 189
0 votes
0 answers
13 views

How to properly mask MultiHeadAttention for sliding window time series data

I have data in the shape (batch, seq_len, features) that is a time series sliding window. In essence, I'm using the most recent ...
user avatar
1 vote
1 answer
35 views

How to obtain Key, Value and Query in Attention and Multi-Head-Attention

I am currently trying to get the hang of BERT and Transformers, so I worked through the Paper "Attention Is All You Need". Now I have a hard time understanding how the Key-, Value-, and ...
user avatar
  • 121
0 votes
2 answers
10 views

Do value and key of additive attention need to have the same dimension?

For the documented tensorflow-keras implementation of additive attention, it is stated that the input tensors are: query: Query Tensor of shape ...
user avatar
  • 101
3 votes
2 answers
41 views

About basic understanding of attention mechanism and model weights

In the image domain, for a given image, suppose that we want to understand if there is a bird on that image (0 and 1 label = no bird and bird), attention mechanism helps to pay more attention to the ...
user avatar
  • 189
0 votes
0 answers
13 views

Extract Local Attention Positions

I am working on a project, I want to extract the local attention positions present in the feature maps of every level in the FPN. Below is the visualization: I am using MMDetection toolbox for ...
user avatar
0 votes
1 answer
46 views

Trainable weights in attention mechanism

I am wondering what are the trainable weights inside an attention-powered transformer. I figure the feed-forward layer contain trainable weights and the token embeddings, but what other parts contain ...
user avatar
6 votes
1 answer
810 views

Why are residual connections needed in transformer architectures?

Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training. This problem is ...
user avatar
1 vote
1 answer
21 views

Why the sequence of matrix transformations in The Annotated Transformer?

Forgive me the long introduction, but I want to be very specific with my question. The Annotated Transformer implements the Transformer architecture from "Attention Is All You Need". The <...
user avatar
  • 11
2 votes
1 answer
50 views

Hard attention derivations

I am trying to completely understand the paper Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. I understand the paper conceptually. I am trying to understand the math ...
user avatar
1 vote
0 answers
25 views

How can one feed all of the input to an RNN, and then get all of the output from it?

When reading papers, a common concept is delaying the output of RNNs to after seeing all of the input. E.g., the neural Turing machine paper uses this technique, together with a simple identity ...
user avatar
  • 121
0 votes
0 answers
26 views

How are the self-attention layer weigths updated?

I am trying to figure out how the updating of the weights in the self attention layer works. I think I have some basic understanding of how the self-attention mechanism works, however it is really ...
user avatar
  • 121
0 votes
1 answer
269 views

Literature to understand the components of Temporal Fusion Transformer

I'm currently reading the paper Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting: https://arxiv.org/abs/1912.09363v3 However, I had to stop at page 7/8, which ...
user avatar
0 votes
0 answers
31 views

Issue related to Transformer decoder druing Inference using all previous output tokens until each decoder time step

I've been trying to understand the shapes used during decoder (both self-attention and enc-dec-attention blocks) and understand there is a difference in the way decoder runs during training versus ...
user avatar
  • 309
0 votes
0 answers
8 views

Why do faster (eg sparse) versions of Transformers focus on the query-key product?

A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism: $ softmax(\frac{Q K^T}{\sqrt{d}}) V $, As I understand it, the runtime, assuming $\{Q, K,...
user avatar
1 vote
0 answers
15 views

In attention models, why is dropout used for positional encodings?

In the "Attention is all you need" paper, they write: we apply dropout to the sums of the embeddings and the positional encodings I can understand why you might use dropout on the ...
user avatar
3 votes
1 answer
47 views

Operation modes in neural turing machine (Graves, 2014)

I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out. More ...
user avatar
  • 75
0 votes
1 answer
36 views

What is purpose of multihead in transformers

Why in the attention mechanism do they apply multihead? It's been said that it would lift the amplitude of vector dot production from word that at the moment is analysed, which can be correct, but ...
user avatar
0 votes
0 answers
79 views

Why transformer in deep learning is called transformer?

Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
user avatar
0 votes
1 answer
127 views

Why the asymmetric design between (Q, K) and V in tranformer's attention?

In the Attention is all you need paper, the self-attention layer is defined as $\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)V$. I would like to know why a more ...
user avatar
0 votes
0 answers
34 views

Recommendation for prediction of multivariate time series

I am looking for a recommendation (concrete package/framework/approach etc., preferably in Python/Keras or R as that is where I have experience) for predicting multivariate time series. I do have ...
user avatar
2 votes
1 answer
29 views

In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?

So attention and transformer models can be used for machine translation. Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
user avatar
  • 365
0 votes
1 answer
173 views

How does tf.keras.MultiHeadAttention layer handle positional encoding?

In Attention Is All You Need paper, positional encodings are added to the input embeddings in order to consider the order of the sequence. How does tf.keras.MultiHeadAttention handle positional ...
user avatar
1 vote
1 answer
125 views

Do we need Masking in test phase while we use transformer?

We use two types of masks when we train transformer models one is in the architecture of encoder to adjust for the length of the input sequence and another is the mask that is being used by the ...
user avatar
0 votes
0 answers
27 views

Attention Scores in Seq2seq sorting example

This question is about the interpretation of the additive attention (Bahdanau) scores in a seq2seq problem concerning the sorting of an input numerical sequence. I have a LSTM Encoder/Decoder model ...
user avatar
  • 1
0 votes
0 answers
16 views

Does swapping Query and Value in Attention make sense here?

I'm building a seq2seq transformer model that uses several convolutional-attention layers in both the encoder and the decoder. The CNN Attention layer takes an input and a 'state' and produces an ...
user avatar
  • 884
0 votes
0 answers
218 views

R Keras attention for 2-D timeseries in a classification task

I don't understand how a regular attention mechanism can be plugged in inside a neural architecture for a two-dimensional (a 2-D tensor) timeseries classification. Let's say, my timeseries has length <...
user avatar
0 votes
0 answers
43 views

Can the positional encoding in the transformer be called a 'stochastic' encoding?

I just want to make sure my intuition is correct in understanding how the positional encodings in "Attention Is All You Need" (https://arxiv.org/pdf/1706.03762.pdf) works... Let the hidden ...
user avatar
  • 619
0 votes
0 answers
34 views

split in linear in multi-head attention

i just learned about transformer and until today i still got confused about somethings. after reading this article one and two there are 2 things i dont understand if in the case is 8 head attention, ...
user avatar
  • 1
0 votes
0 answers
221 views

The last question about query-key-value matrices in Transformer

There are a lot of questions regarding query-key-value concept in Transformers, however, the concept and the details are still unclear (at least for me). Here I will describe my questions in details, ...
user avatar
  • 265
0 votes
0 answers
118 views

Transformer FF-Network output and encoder-decoder attention

I am trying to follow through Jay Alammar's blogpost on transformers, but cannot quite grasp all the details. Encoder-Attention: So, as to my understanding, we are given eight different Query/Key/...
user avatar
  • 1
3 votes
0 answers
110 views

Why PyTorch MultiheadAttention is considered as activation function?

When I scroll all activation functions available on PyTorch package (here), I found that nn.MultiheadAttention is described there. Can you please explain why it's ...
user avatar
  • 131
2 votes
1 answer
305 views

Why is Bahdanau's attention sometimes called concat attention?

I am learning the intuition behind the attention mechanism from https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/ https://lilianweng.github....
user avatar
  • 155
0 votes
0 answers
101 views

Can the encoder and decoder have different hidden dimensions in the Transformer architecture?

In the vanilla Transformer architecture, can we have an encoder with hidden size of 768, and a decoder with hidden size of 792?
user avatar
1 vote
1 answer
295 views

How pre-trained weights in the BERT can help the fine tuning task?

I have been using the BERT architecture implemented by the Huggingface library for my sentence classification task. Although, I read the paper (and related papers) and the result of my experiments is ...
user avatar
  • 275
1 vote
1 answer
146 views

How does a transformer network's attention mechanism determine what to pay attention to?

The paper Attention is All You Need describes the transformer network which uses a "multi-head attention mechanism". The basic intuition behind this mechanism is that it weighs other tokens ...
user avatar
4 votes
1 answer
339 views

When calculating self-attention for Transformer ML architectures, why do we need both a key and a query weight matrix?

I'm trying to understand the math behind Transformers, specifically self-attention. This link, and many others, gives the formula to compute the output vectors from the input embeddings as: $$Q=XW_Q,\;...
user avatar
  • 43
6 votes
1 answer
3k views

Deciding between Decoder-only or Encoder-only Transformers (BERT, GPT)

I just started learning about transformers and looked into the following 3 variants The original one from Attention Is All You Need (Encoder & Decoder) BERT (Encoder only) GPT-2 (Decoder only) ...
user avatar
1 vote
3 answers
788 views

transformer, linear layer after attention

My question regards this image: It seems that after the multi head attention there is a linear layer as they mention also from here: the linearity is given by the weights W^{o}. my quesion is: for ...
user avatar
  • 13
2 votes
0 answers
92 views

Attention dropout, where was it proposed/used first?

Attention dropout (dropout on the attention weights) is very common for the Transformer model. In the original Attention is all you need paper, dropout is mentioned, but not for the attention weights. ...
user avatar
  • 1,155
2 votes
1 answer
1k views

What is masking in the attention if all you need paper?

I am a newbie to the NLP and specifically, the attention is all you need and I can understand the encoder part of the paper. However, I am baffled about the decoder part. In the pic below and the ...
user avatar
  • 21
0 votes
1 answer
20 views

Attention is all you need: During run/test time, when output is not available, how is the decoder used? [closed]

In the famous paper "Attention is all you need" we see that in the Decoder we input the supposedly 'Output' sentence embeddings. During inference/test time, this output would not be ...
user avatar
2 votes
0 answers
272 views

In the attention mechanism, why are there separate weight matrices for the queries and keys?

To perform self attention over a collection of $n$ vectors stacked up into a matrix $X \in \mathbb{R}^{n \times d}$, we first obtain query, key, and value representations of these vectors via three ...
user avatar
  • 3,205
0 votes
1 answer
50 views

Attention is all you need: During training, does decoder generates the whole sentence in 1 shot and not sequentially word by word?

In the famous paper "Attention is all you need", does the decoder generates the whole output sentence in one shot in parallel. In this case will the final softmax output the probabilities ...
user avatar
0 votes
1 answer
57 views

Attention is all you need: from where does it get the encoder/decoder input embeddings?

In "Attention is all you need" paper, regarding encoder (and decoder) input embeddings: Do they use already pretrained such as off the shelf Word2vec or Glove embeddings ? or are they also ...
user avatar
1 vote
0 answers
41 views

Any natural interpretation for "hopfield network is all you need"?

I am studying deep learning and came across to the paper claiming "so-called attention mechanism in deep learning is equivalent to hopfield network", in the paper called "Hopfield ...
user avatar
  • 139
2 votes
1 answer
391 views

Why use sinusoidal along embedding dimension in positional encoding in transformers?

Why do we need sinusoidal function along the embedding dimension in positional encoding in transformers? Shouldn't sinusoidal function along time dimension be enough? This question is derived from ...
user avatar
  • 25
2 votes
1 answer
825 views

Does it make sense to use attention mechanism for seq-2-seq autoencoder for anomaly detection?

So I want to train LSTM sequence to sequence model, autoencoder, for anomaly detection. The idea is to train it on normal samples and when anomaly comes into model it will not be able to reconstruct ...
user avatar
  • 731