Questions tagged [lstm]

A Long Short Term Memory (LSTM) is a neural network architecture that contains recurrent NN blocks that can remember a value for an arbitrary length of time.

Filter by
Sorted by
Tagged with
2
votes
1answer
1k views

Fastest LSTM training in CPU-only (no-GPU) setting [closed]

Training more or less sophisticated RNNs (e.g. LSTM) takes ages in CPU-only setting. I've tried Lasagne-over-Theano, Keras-over-TF and torch-rnn implementations of character-level language models (...
33
votes
1answer
54k views

Training loss goes down and up again. What is happening?

My training loss goes down and then up again. It is very weird. The cross-validation loss tracks the training loss. What is going on? I have two stacked LSTMS as follows (on Keras): ...
4
votes
1answer
8k views

Keras - LSTM: need for a final dense layer

I am looking at the text generation example using Keras here and I noticed that a Dense(len(chars)) is included as the last layer. Given that LSTM itself can ...
0
votes
1answer
2k views

How to use LSTM as a sequence classifier?

I have got the following problem at hand. I have variable length videos which belong to one of the four classes $A,B,C,D$. From each frame of a video, I extract a feature vector of length $N$. ...
2
votes
1answer
692 views

LSTM: doesn't forget gate interfere to input gate?

I need to implement an LSTM, so I'm reading a tutorial of how does it work. Quoted (the relevant parts; if you're aware of LSTM internals, you're probably want to just skip down to the question, and ...
3
votes
1answer
902 views

Using an RNN/LSTM to generate sequences with a unique output

I'm trying to train a LSTM recurrent neural network where my data consists of a sequence of animal migration data ...
5
votes
1answer
1k views

What is the purpose of the scaling factor used in dropout?

I have a question related to the dropout function in the LSTM tutorial: http://deeplearning.net/tutorial/code/lstm.py ...
8
votes
2answers
5k views

What happens when we feed a 2D matrix to a LSTM layer

Suppose I am feeding a 2D matrix of shape (99,13) as input to a LSTM layer. I am having n number of files, where each contains (99,13) size vectors. I have decided to consider 13 as the number of ...
8
votes
1answer
5k views

Variable importance in RNN or LSTM

Several method have been devised for accessing or quantifying variable importance (even if only relative to each other) in MLP neural network models: Connection weights Garson’s algorithm Partial ...
0
votes
1answer
424 views

Translating a TensorFlow LSTM into synapticjs

I'm working on implementing an interface between a TensorFlow basic LSTM that's already been trained and a javascript version that can be run in the browser. The problem is that in all of the ...
9
votes
2answers
6k views

Best use of LSTM for within sequence event prediction

Assume the following 1 dimensional sequence: A, B, C, Z, B, B, #, C, C, C, V, $, W, A, % ... Letters A, B, C, .. here ...
37
votes
4answers
22k views

How does LSTM prevent the vanishing gradient problem?

LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from Greff et al.) correspond to ...
1
vote
1answer
211 views

LSTM Classifying all the words as the same class

I've used Lasagne to build a LSTM model to classify words with the IOB-tags. About 25-40% of the training words classes is O, thus receiving the same int32 class number 126. The words go through a ...
10
votes
3answers
11k views

Structure of Recurrent Neural Network (LSTM, GRU)

I am trying to understand the architecture of RNNs. I have found this tutorial which has been very helpful: http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Especially this image: How ...
23
votes
2answers
15k views

What optimization methods work best for LSTMs?

I've been using theano to experiment with LSTMs, and was wondering what optimization methods (SGD, Adagrad, Adadelta, RMSprop, Adam, etc) work best for LSTMs? Are there any research papers on this ...
3
votes
2answers
182 views

Question on Hochreiter's LSTM paper

On the bottom of page 3 of the LSTM paper, it says "This will scale the error by the following factor" $$\frac{\partial \vartheta_v(t-q)}{\partial\vartheta_u(t)}=\cdots$$ This is very strange to me, ...
32
votes
4answers
16k views

What are the advantages of stacking multiple LSTMs?

What are the advantages, why would one use multiple LSTMs, stacked one side-by-side, in a deep-network? I am using a LSTM to represent a sequence of inputs as a single input. So once I have that ...
6
votes
1answer
1k views

Training LSTM a sequence one item at a time

I am trying to train an lstm with a sequence and get the sequence classification for the whole sequence. I have sequences of varying length so I have one input neuron and I am feeding one item at a ...
13
votes
1answer
23k views

How to train LSTM layer of deep-network

I'm using a lstm and feed-forward network to classify text. I convert the text into one-hot vectors and feed each into the lstm so I can summarise it as a single representation. Then I feed it to the ...
19
votes
3answers
34k views

Using RNN (LSTM) for predicting the timeseries vectors (Theano)

I have very simple problem but I cannot find a right tool to solve it. I have some sequence of vectors of the same length. Now I would like to train LSTM RNN on train sample of these sequences and ...
19
votes
1answer
21k views

What is a feasible sequence length for an RNN to model?

I'm looking into using a LSTM (long short-term memory) version of a recurrent neural network (RNN) for modeling timeseries data. As the sequence length of the data increases, the complexity of the ...
7
votes
1answer
1k views

Why use a mixture model with RNN instead of just directly predictive real values?

Alex Graves created a model to generate hand writing sequences which use an LSTM (kind of Recurrent Neural Network) to predict the parameters for an mixture model. The mixture model is then used to ...
5
votes
2answers
647 views

LSTM forgetting dependencies

How does a LSTM network know when is a good time to forget the dependencies it has learned? I understand that it forgets when the value of forget gate is close to zero. But how does it know when to ...
3
votes
0answers
202 views

Hessian-Free instead of LSTM for Recurrent Net Machine Translation

Last year, Ilya Sutskever and collaborators came out with a paper about a recurrent LSTM net that learns sequence to sequence mappings for machine translation. It's somewhat surprising that the ...
3
votes
1answer
468 views

Stackable LSTM layer trained with arbitrary BPTT time steps

Anyone knows how to make a LSTM layer that is able to be trained with arbitrary BPTT time steps and easy to be stacked together? I am now implementing a basic version of LSTM layer. My scan function ...

1
9 10 11 12
13