Questions tagged [attention]
Methods for aggregating a set feature vectors into a single feature vector relevant to a context.
22
questions
1
vote
2answers
84 views
What exactly are keys, queries, and values in attention mechanisms?
How should one understand the keys, queries, and values that are often mentioned in attention mechanisms?
I've tried searching online, but all the resources I find only speak of them as if the reader ...
0
votes
0answers
18 views
Self-attention original work? [duplicate]
I'm curious if there's an original research paper regarding the self-attention mechanism. I find it difficult to understand the entire "query-key-value" mechanism that is frequently used to describe ...
1
vote
1answer
142 views
Is it true that Bahdanau's attention mechanism is not Global like Luong's?
I was reading the pytorch tutorial on a chatbot task and attention where it said:
Luong et al. improved upon Bahdanau et al.’s groundwork by creating “Global attention”. The key difference is that ...
3
votes
2answers
173 views
Why do attention models need to choose a maximum sentence length?
I was going through the seq2seq-translation tutorial on pytorch and found the following sentence:
Because there are sentences of all sizes in the training data, to actually create and train this ...
0
votes
1answer
34 views
Attention weights for identifying influence
This question concerns the application of self-attention weights for identifying influence of words in sentences.
For instance, we are performing a classification task on a set of sentences (e.g. ...
0
votes
0answers
48 views
Attention mechanism in LSTM model to predict next action
I have a dataset. Each data point of this set contains a variable length of sequences with 7 letters. For example, Data point 1 has a sequence (A, A, B, E, B, C, D, E, D.....). I used LSTM to predict ...
0
votes
0answers
26 views
performing function (maxcol) across the column
I'm currently going through this paper: Bidirectional Attention Flow for Machine Comprehension, Seo, Minjoon, et al. (2016)
They perform a $max_{col}$ function over a matrix $S \in \mathbb R^{TxJ}$:
...
0
votes
2answers
61 views
Encoding Layers in the Transformer
In the transformer architecture for NLP, at each layer there are multiple self-attention filters. My question is about the encoded content within these filters. An example can be found here:
My ...
0
votes
1answer
50 views
How to use/treat a hidden layer as the new target to predict in a neural network?
Let's suppose I have a neural network with one hidden layer. During training, for a given pair of (input, target), I want to perform several iterations, such that the first iteration would be trying ...
0
votes
1answer
44 views
Attention for short sequence length. Is it reasonable?
Will the attention mechanism be useful for the short sequence length?
Let's say your training corpus has each query of MAX length 10. and most queries are of word length 3-4 words. How reasonable is ...
1
vote
1answer
181 views
Hard attention loss function
I am referring to paper: Show, Attend and Tell: Neural Image Caption
Generation with Visual Attention (page 4). I wished to know, why we look to maximize the lower bound of the log likelihood ...
0
votes
1answer
28 views
How does network structure (model complexity) affects covergence speed?
I trained Bi-GRU and HAN (Hierarchical Attention Networks) on my own datasets, and found HAN converges faster than Bi-GRU, within less number of epochs.
What would be the reason for this? I guess ...
2
votes
1answer
813 views
On masked multi-head attention and layer normalization in transformer model
I came to read Attention is All you Need by Vaswani. There two questions came up to me:
1. How is it possible to mask out illegal connections in decoder multi-head attention?
It says by setting ...
2
votes
1answer
124 views
Custom Loss Function - Inducing sparsity
From the comments, I realized that my question wasn't clear enough, so I'll start with a short background.
I am trying to construct an attention model that performs classification based on just a ...
0
votes
1answer
218 views
Can attention be implemented without encoder / decoder?
I just got into models beyond biLSTM, would like to start with applying attention to my existing network (RNN). I find examples for attention always with encoder decoder architecture, however is it ...
0
votes
1answer
84 views
What are paralell training and attention mechanism?
I read a quite interesting paper here:
http://hanj.cs.illinois.edu/pdf/kdd18_cyang.pdf
Accordingly, the basic idea is to combine clustering and churn prediction so that it can imply some insight from ...
1
vote
1answer
615 views
How to use the transformer for inference
I am trying to understand the transformer model from Attention is all you need, following the annotated transformer.
The architecture looks like this:
Everything is essentially clear, save for the ...
5
votes
2answers
218 views
What stops the network from learning the same weights in multi-head attention mechanism
I have been trying to understand the transformer network and specifically the multi-head attention bit. So, as I understand it that multiple attention weighted linear combination of the input features ...
1
vote
1answer
101 views
Attention based models for machine comprehension
Currently, I am understanding Attention models. I specifically need it to build a machine comprehension model (a model which can find answer to a question from a given comprehension). But I want to ...
0
votes
1answer
39 views
Meaning of different softmax notations in papers
I was wondering if the different notations of the softmax input mean different things especially about the size of the output. For example, in the paper Pointer Networks, it sometimes state the input ...
22
votes
1answer
3k views
What are attention mechanisms exactly?
Attention mechanisms have been used in various Deep Learning papers in the last few years. Ilya Sutskever, head of research at Open AI, has enthusiastically praised them:
https://towardsdatascience....
5
votes
1answer
208 views
Attention methods
When using Attention, for example with LSTM (but not necessarily), one can use the following methods to attend:
MLP: $ug(W^1v+W^2q)$
dot product: $v \cdot q$
biaffine transform: $v^TWq$
($v$ is the ...
