Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [attention]

Methods for aggregating a set feature vectors into a single feature vector relevant to a context.

0
votes
0answers
14 views

Attention weights for identifying influence

This question concerns the application of self-attention weights for identifying influence of words in sentences. For instance, we are performing a classification task on a set of sentences (e.g. ...
0
votes
0answers
29 views

Attention mechanism in LSTM model to predict next action

I have a dataset. Each data point of this set contains a variable length of sequences with 7 letters. For example, Data point 1 has a sequence (A, A, B, E, B, C, D, E, D.....). I used LSTM to predict ...
0
votes
0answers
25 views

performing function (maxcol) across the column

I'm currently going through this paper: Bidirectional Attention Flow for Machine Comprehension, Seo, Minjoon, et al. (2016) They perform a $max_{col}$ function over a matrix $S \in \mathbb R^{TxJ}$: ...
0
votes
1answer
24 views

Encoding Layers in the Transformer

In the transformer architecture for NLP, at each layer there are multiple self-attention filters. My question is about the encoded content within these filters. An example can be found here: My ...
0
votes
1answer
49 views

How to use/treat a hidden layer as the new target to predict in a neural network?

Let's suppose I have a neural network with one hidden layer. During training, for a given pair of (input, target), I want to perform several iterations, such that the first iteration would be trying ...
0
votes
1answer
32 views

Attention for short sequence length. Is it reasonable?

Will the attention mechanism be useful for the short sequence length? Let's say your training corpus has each query of MAX length 10. and most queries are of word length 3-4 words. How reasonable is ...
1
vote
1answer
101 views

Hard attention loss function

I am referring to paper: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (page 4). I wished to know, why we look to maximize the lower bound of the log likelihood ...
0
votes
1answer
19 views

How does network structure (model complexity) affects covergence speed?

I trained Bi-GRU and HAN (Hierarchical Attention Networks) on my own datasets, and found HAN converges faster than Bi-GRU, within less number of epochs. What would be the reason for this? I guess ...
1
vote
1answer
265 views

On masked multi-head attention and layer normalization in transformer model

I came to read Attention is All you Need by Vaswani. There two questions came up to me: 1. How is it possible to mask out illegal connections in decoder multi-head attention? It says by setting ...
2
votes
1answer
106 views

Custom Loss Function - Inducing sparsity

From the comments, I realized that my question wasn't clear enough, so I'll start with a short background. I am trying to construct an attention model that performs classification based on just a ...
0
votes
1answer
124 views

Can attention be implemented without encoder / decoder?

I just got into models beyond biLSTM, would like to start with applying attention to my existing network (RNN). I find examples for attention always with encoder decoder architecture, however is it ...
0
votes
1answer
77 views

What are paralell training and attention mechanism?

I read a quite interesting paper here: http://hanj.cs.illinois.edu/pdf/kdd18_cyang.pdf Accordingly, the basic idea is to combine clustering and churn prediction so that it can imply some insight from ...
1
vote
1answer
293 views

How to use the transformer for inference

I am trying to understand the transformer model from Attention is all you need, following the annotated transformer. The architecture looks like this: Everything is essentially clear, save for the ...
3
votes
2answers
136 views

What stops the network from learning the same weights in multi-head attention mechanism

I have been trying to understand the transformer network and specifically the multi-head attention bit. So, as I understand it that multiple attention weighted linear combination of the input features ...
1
vote
1answer
91 views

Attention based models for machine comprehension

Currently, I am understanding Attention models. I specifically need it to build a machine comprehension model (a model which can find answer to a question from a given comprehension). But I want to ...
0
votes
1answer
31 views

Meaning of different softmax notations in papers

I was wondering if the different notations of the softmax input mean different things especially about the size of the output. For example, in the paper Pointer Networks, it sometimes state the input ...
21
votes
1answer
2k views

What are attention mechanisms exactly?

Attention mechanisms have been used in various Deep Learning papers in the last few years. Ilya Sutskever, head of research at Open AI, has enthusiastically praised them: https://towardsdatascience....
5
votes
1answer
183 views

Attention methods

When using Attention, for example with LSTM (but not necessarily), one can use the following methods to attend: MLP: $ug(W^1v+W^2q)$ dot product: $v \cdot q$ biaffine transform: $v^TWq$ ($v$ is the ...