Questions tagged [attention]
Methods for aggregating a set feature vectors into a single feature vector relevant to a context.
0
votes
0answers
14 views
Attention weights for identifying influence
This question concerns the application of self-attention weights for identifying influence of words in sentences.
For instance, we are performing a classification task on a set of sentences (e.g. ...
0
votes
0answers
29 views
Attention mechanism in LSTM model to predict next action
I have a dataset. Each data point of this set contains a variable length of sequences with 7 letters. For example, Data point 1 has a sequence (A, A, B, E, B, C, D, E, D.....). I used LSTM to predict ...
0
votes
0answers
25 views
performing function (maxcol) across the column
I'm currently going through this paper: Bidirectional Attention Flow for Machine Comprehension, Seo, Minjoon, et al. (2016)
They perform a $max_{col}$ function over a matrix $S \in \mathbb R^{TxJ}$:
...
0
votes
1answer
24 views
Encoding Layers in the Transformer
In the transformer architecture for NLP, at each layer there are multiple self-attention filters. My question is about the encoded content within these filters. An example can be found here:
My ...
0
votes
1answer
49 views
How to use/treat a hidden layer as the new target to predict in a neural network?
Let's suppose I have a neural network with one hidden layer. During training, for a given pair of (input, target), I want to perform several iterations, such that the first iteration would be trying ...
0
votes
1answer
32 views
Attention for short sequence length. Is it reasonable?
Will the attention mechanism be useful for the short sequence length?
Let's say your training corpus has each query of MAX length 10. and most queries are of word length 3-4 words. How reasonable is ...
1
vote
1answer
101 views
Hard attention loss function
I am referring to paper: Show, Attend and Tell: Neural Image Caption
Generation with Visual Attention (page 4). I wished to know, why we look to maximize the lower bound of the log likelihood ...
0
votes
1answer
19 views
How does network structure (model complexity) affects covergence speed?
I trained Bi-GRU and HAN (Hierarchical Attention Networks) on my own datasets, and found HAN converges faster than Bi-GRU, within less number of epochs.
What would be the reason for this? I guess ...
1
vote
1answer
265 views
On masked multi-head attention and layer normalization in transformer model
I came to read Attention is All you Need by Vaswani. There two questions came up to me:
1. How is it possible to mask out illegal connections in decoder multi-head attention?
It says by setting ...
2
votes
1answer
106 views
Custom Loss Function - Inducing sparsity
From the comments, I realized that my question wasn't clear enough, so I'll start with a short background.
I am trying to construct an attention model that performs classification based on just a ...
0
votes
1answer
124 views
Can attention be implemented without encoder / decoder?
I just got into models beyond biLSTM, would like to start with applying attention to my existing network (RNN). I find examples for attention always with encoder decoder architecture, however is it ...
0
votes
1answer
77 views
What are paralell training and attention mechanism?
I read a quite interesting paper here:
http://hanj.cs.illinois.edu/pdf/kdd18_cyang.pdf
Accordingly, the basic idea is to combine clustering and churn prediction so that it can imply some insight from ...
1
vote
1answer
293 views
How to use the transformer for inference
I am trying to understand the transformer model from Attention is all you need, following the annotated transformer.
The architecture looks like this:
Everything is essentially clear, save for the ...
3
votes
2answers
136 views
What stops the network from learning the same weights in multi-head attention mechanism
I have been trying to understand the transformer network and specifically the multi-head attention bit. So, as I understand it that multiple attention weighted linear combination of the input features ...
1
vote
1answer
91 views
Attention based models for machine comprehension
Currently, I am understanding Attention models. I specifically need it to build a machine comprehension model (a model which can find answer to a question from a given comprehension). But I want to ...
0
votes
1answer
31 views
Meaning of different softmax notations in papers
I was wondering if the different notations of the softmax input mean different things especially about the size of the output. For example, in the paper Pointer Networks, it sometimes state the input ...
21
votes
1answer
2k views
What are attention mechanisms exactly?
Attention mechanisms have been used in various Deep Learning papers in the last few years. Ilya Sutskever, head of research at Open AI, has enthusiastically praised them:
https://towardsdatascience....
5
votes
1answer
183 views
Attention methods
When using Attention, for example with LSTM (but not necessarily), one can use the following methods to attend:
MLP: $ug(W^1v+W^2q)$
dot product: $v \cdot q$
biaffine transform: $v^TWq$
($v$ is the ...
