Questions tagged [attention]
Methods for aggregating a set feature vectors into a single feature vector relevant to a context.
30
questions
1
vote
1answer
11 views
What is the target variable in the feed forward neural network within the Transformer model architecture?
In the paper 'Attention is all you need' the model architecture of The Transformer is described. Both in the encoder as well as in the decoder, there is a feed forward network. If I understant it ...
1
vote
1answer
62 views
Why is local-m Attention differentiable?
In this paper (https://arxiv.org/pdf/1508.04025.pdf) local attention is introduced. With local-m attention, we compute the attention vector over the source hidden states within a window around the ...
0
votes
1answer
22 views
Need of value vector in transformers
I am reading the paper "Attention is all you need" (https://arxiv.org/abs/1706.03762). In transformer architecture, we have 3 vectors(key,value and query) for each word. I don't understand the need of ...
0
votes
2answers
35 views
How to account for the no:of parameters in the Multihead self-Attention layer of BERT
I have read the BERT paper for NLP (https://arxiv.org/abs/1810.04805), and trying to understand the keras implementation. Here's my code to load the BERT model:
...
0
votes
1answer
11 views
Visualine Node Importance in Graph Convolution
I am quite new to the concept of attention. I am working with graph data and running graph convolution on it to learn node level embedding first. Then an attention layer to aggregate the nodes to ...
0
votes
1answer
88 views
Why use both $\sin$ and $\cos$ functions in Transformer positional encoding?
I am interested why the positional encoding of Transformer use both $\sin$ and $\cos$, I understand use only $\sin$ will make all dimension equals to 0 in position 0.
But $\cos$ don't have such ...
3
votes
1answer
87 views
Why K and V are not the same in Transformer attention?
My understanding is for translation task K should be the same with V, but in Transformer K and V are generated by two different(randomly initialized) matrix $W^K, W^V$, therefore not the same. Can any ...
0
votes
0answers
24 views
The meaning of distance measure in Additive attention and Dot Product Attention
Currently, I am studying about Transformer (reading this paper; attention is all you need). In the paper, the author said,
(3.2.1) Additive attention computes the compatibility function using a ...
14
votes
3answers
3k views
What exactly are keys, queries, and values in attention mechanisms?
How should one understand the keys, queries, and values that are often mentioned in attention mechanisms?
I've tried searching online, but all the resources I find only speak of them as if the reader ...
0
votes
0answers
158 views
Self-attention original work? [duplicate]
I'm curious if there's an original research paper regarding the self-attention mechanism. I find it difficult to understand the entire "query-key-value" mechanism that is frequently used to describe ...
1
vote
1answer
306 views
Is it true that Bahdanau's attention mechanism is not Global like Luong's?
I was reading the pytorch tutorial on a chatbot task and attention where it said:
Luong et al. improved upon Bahdanau et al.’s groundwork by creating “Global attention”. The key difference is that ...
4
votes
2answers
318 views
Why do attention models need to choose a maximum sentence length?
I was going through the seq2seq-translation tutorial on pytorch and found the following sentence:
Because there are sentences of all sizes in the training data, to actually create and train this ...
0
votes
1answer
64 views
Attention weights for identifying influence
This question concerns the application of self-attention weights for identifying influence of words in sentences.
For instance, we are performing a classification task on a set of sentences (e.g. ...
0
votes
0answers
55 views
Attention mechanism in LSTM model to predict next action
I have a dataset. Each data point of this set contains a variable length of sequences with 7 letters. For example, Data point 1 has a sequence (A, A, B, E, B, C, D, E, D.....). I used LSTM to predict ...
0
votes
0answers
30 views
performing function (maxcol) across the column
I'm currently going through this paper: Bidirectional Attention Flow for Machine Comprehension, Seo, Minjoon, et al. (2016)
They perform a $max_{col}$ function over a matrix $S \in \mathbb R^{TxJ}$:
...
1
vote
2answers
86 views
Encoding Layers in the Transformer
In the transformer architecture for NLP, at each layer there are multiple self-attention filters. My question is about the encoded content within these filters. An example can be found here:
My ...
1
vote
1answer
55 views
How to use/treat a hidden layer as the new target to predict in a neural network?
Let's suppose I have a neural network with one hidden layer. During training, for a given pair of (input, target), I want to perform several iterations, such that the first iteration would be trying ...
0
votes
1answer
53 views
Attention for short sequence length. Is it reasonable?
Will the attention mechanism be useful for the short sequence length?
Let's say your training corpus has each query of MAX length 10. and most queries are of word length 3-4 words. How reasonable is ...
1
vote
1answer
251 views
Hard attention loss function
I am referring to paper: Show, Attend and Tell: Neural Image Caption
Generation with Visual Attention (page 4). I wished to know, why we look to maximize the lower bound of the log likelihood ...
0
votes
1answer
33 views
How does network structure (model complexity) affects covergence speed?
I trained Bi-GRU and HAN (Hierarchical Attention Networks) on my own datasets, and found HAN converges faster than Bi-GRU, within less number of epochs.
What would be the reason for this? I guess ...
3
votes
1answer
2k views
On masked multi-head attention and layer normalization in transformer model
I came to read Attention is All you Need by Vaswani. There two questions came up to me:
1. How is it possible to mask out illegal connections in decoder multi-head attention?
It says by setting ...
2
votes
1answer
157 views
Custom Loss Function - Inducing sparsity
From the comments, I realized that my question wasn't clear enough, so I'll start with a short background.
I am trying to construct an attention model that performs classification based on just a ...
0
votes
1answer
289 views
Can attention be implemented without encoder / decoder?
I just got into models beyond biLSTM, would like to start with applying attention to my existing network (RNN). I find examples for attention always with encoder decoder architecture, however is it ...
0
votes
1answer
88 views
What are paralell training and attention mechanism?
I read a quite interesting paper here:
http://hanj.cs.illinois.edu/pdf/kdd18_cyang.pdf
Accordingly, the basic idea is to combine clustering and churn prediction so that it can imply some insight from ...
2
votes
1answer
983 views
How to use the transformer for inference
I am trying to understand the transformer model from Attention is all you need, following the annotated transformer.
The architecture looks like this:
Everything is essentially clear, save for the ...
5
votes
2answers
428 views
What stops the network from learning the same weights in multi-head attention mechanism
I have been trying to understand the transformer network and specifically the multi-head attention bit. So, as I understand it that multiple attention weighted linear combination of the input features ...
1
vote
1answer
106 views
Attention based models for machine comprehension
Currently, I am understanding Attention models. I specifically need it to build a machine comprehension model (a model which can find answer to a question from a given comprehension). But I want to ...
1
vote
1answer
52 views
Meaning of different softmax notations in papers
I was wondering if the different notations of the softmax input mean different things especially about the size of the output. For example, in the paper Pointer Networks, it sometimes state the input ...
24
votes
1answer
3k views
What are attention mechanisms exactly?
Attention mechanisms have been used in various Deep Learning papers in the last few years. Ilya Sutskever, head of research at Open AI, has enthusiastically praised them:
https://towardsdatascience....
5
votes
1answer
224 views
Attention methods
When using Attention, for example with LSTM (but not necessarily), one can use the following methods to attend:
MLP: $ug(W^1v+W^2q)$
dot product: $v \cdot q$
biaffine transform: $v^TWq$
($v$ is the ...
