Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [lstm]

A Long Short Term Memory (LSTM) is a neural network architecture that contains recurrent NN blocks that can remember a value for an arbitrary length of time.

30
votes
3answers
14k views

How does LSTM prevent the vanishing gradient problem?

LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from Greff et al.) correspond to ...
28
votes
4answers
17k views

Understanding LSTM units vs. cells

I have been studying LSTMs for a while. I understand at a high level how everything works. However, going to implement them using Tensorflow I've noticed that BasicLSTMCell requires a number of units (...
23
votes
1answer
36k views

Training loss goes down and up again. What is happening?

My training loss goes down and then up again. It is very weird. The cross-validation loss tracks the training loss. What is going on? I have two stacked LSTMS as follows (on Keras): ...
21
votes
1answer
2k views

What are attention mechanisms exactly?

Attention mechanisms have been used in various Deep Learning papers in the last few years. Ilya Sutskever, head of research at Open AI, has enthusiastically praised them: https://towardsdatascience....
19
votes
3answers
19k views

Understanding input_shape parameter in LSTM with Keras

I'm trying to use the example described in the Keras documentation named "Stacked LSTM for sequence classification" (see code below) and can't figure out the ...
19
votes
3answers
9k views

What are the advantages of stacking multiple LSTMs?

What are the advantages, why would one use multiple LSTMs, stacked one side-by-side, in a deep-network? I am using a LSTM to represent a sequence of inputs as a single input. So once I have that ...
19
votes
3answers
34k views

Using RNN (LSTM) for predicting the timeseries vectors (Theano)

I have very simple problem but I cannot find a right tool to solve it. I have some sequence of vectors of the same length. Now I would like to train LSTM RNN on train sample of these sequences and ...
19
votes
2answers
12k views

What optimization methods work best for LSTMs?

I've been using theano to experiment with LSTMs, and was wondering what optimization methods (SGD, Adagrad, Adadelta, RMSprop, Adam, etc) work best for LSTMs? Are there any research papers on this ...
15
votes
3answers
8k views

Why are the weights of RNN/LSTM networks shared across time?

I've recently become interested in LSTMs and I was surprised to learn that the weights are shared across time. I know that if you share the weights across time, then your input time sequences can ...
15
votes
1answer
15k views

Difference between samples, time steps and features in neural network

I am going through the following blog on LSTM neural network: http://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/ The author reshapes the input ...
14
votes
1answer
15k views

What is a feasible sequence length for an RNN to model?

I'm looking into using a LSTM (long short-term memory) version of a recurrent neural network (RNN) for modeling timeseries data. As the sequence length of the data increases, the complexity of the ...
13
votes
3answers
24k views

Difference between feedback RNN and LSTM/ GRU

I am trying to understand different Recurrent neural network (RNN) architectures to be applied to time series data and I am getting a bit confused with the different names that are frequently used ...
12
votes
1answer
20k views

How to train LSTM layer of deep-network

I'm using a lstm and feed-forward network to classify text. I convert the text into one-hot vectors and feed each into the lstm so I can summarise it as a single representation. Then I feed it to the ...
12
votes
1answer
11k views

Preventing overfitting of LSTM on small dataset

I'm modeling 15000 tweets for sentiment prediction using a single layer LSTM with 128 hidden units using a word2vec-like representation with 80 dimensions. I get a descent accuracy (38% with random = ...
11
votes
2answers
2k views

Why can RNNs with LSTM units also suffer from “exploding gradients”?

I have a basic knowledge of how RNNs (and, in particular, with LSTMs units) work. I have a pictorial idea of the architecture of an LSTM unit, that is a cell and a few gates, which regulate the flow ...
10
votes
2answers
5k views

Best use of LSTM for within sequence event prediction

Assume the following 1 dimensional sequence: A, B, C, Z, B, B, #, C, C, C, V, $, W, A, % ... Letters A, B, C, .. here ...
9
votes
3answers
8k views

Structure of Recurrent Neural Network (LSTM, GRU)

I am trying to understand the architecture of RNNs. I have found this tutorial which has been very helpful: http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Especially this image: How ...
9
votes
1answer
5k views

Time series prediction using ARIMA vs LSTM

The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values....
9
votes
1answer
566 views

Understanding LSTM topology

As many others have, I found the resources here and here to be immensely useful for understanding LSTM cells. I am confident I understand how values flow and are updated, and I'm confident enough to ...
8
votes
1answer
5k views

RNNs: When to apply BPTT and/or update weights?

I am trying to understand the high-level application of RNNs to sequence labeling via (among others) Graves' 2005 paper on phoneme classification. To summarize the problem: We have a large training ...
8
votes
2answers
10k views

What is the output of a tf.nn.dynamic_rnn()?

I am not sure about what I understand from the official documentation, which says: Returns: A pair (outputs, state) where: outputs: The RNN output Tensor. ...
8
votes
2answers
12k views

Handling unknown words in language modeling tasks using LSTM

For a natural language processing (NLP) task one often uses word2vec vectors as an embedding for the words. However, there may be many unknown words that are not captured by the word2vec vectors ...
8
votes
3answers
2k views

RNN for irregular time intervals?

RNNs are remarkably good for capturing the time-dependence of sequential data. However, what happens when the sequence elements aren't equally spaced in time? E.g., the first input to the LSTM cell ...
8
votes
1answer
2k views

Sudden accuracy drop when training LSTM or GRU in Keras

My recurrent neural network (LSTM, resp. GRU) behaves in a way I cannot explain. The training starts and it trains well (the results look quite good) when suddenly accuracy drops (and loss rapidly ...
8
votes
2answers
3k views

What happens when we feed a 2D matrix to a LSTM layer

Suppose I am feeding a 2D matrix of shape (99,13) as input to a LSTM layer. I am having n number of files, where each contains (99,13) size vectors. I have decided to consider 13 as the number of ...
7
votes
2answers
6k views

How to make LSTM predict multiple time steps ahead?

I am trying to use a LSTM for time series prediction. The data streams in once per minute, but I would like to predict an hour ahead. There are two ways I can think of for going about this: Squash ...
7
votes
1answer
3k views

How to train LSTM model on multiple time series data?

How to train LSTM model on multiple time series data? Use case: I have weekly sales of 20,000 agents for last 5 years. Need to forecast upcoming weekly sales for each agent. Do I need to follow a ...
7
votes
3answers
1k views

LSTM time series with mixed frequency data

I want to make a LSTM RNN for timeseries prediction, but some of my predictors are monthly and others are daily. Any advice / examples on how to set up this network? The frequency of the ...
7
votes
2answers
584 views

Why LSTM performs worse in information latching than vanilla recurrent neuron network

I would like to understand better why LSTM can remember information for a longer time period than vanilla/simple recurrent neural network (SRNN) by redoing an experiment from the paper Learning Long-...
7
votes
2answers
685 views

Difference between a single unit LSTM and 3-unit LSTM neural network

The LSTM in the following Keras code input_t = Input((4, 1)) output_t = LSTM(1)(input_t) model = Model(inputs=input_t, outputs=output_t) print(model.summary()) ...
7
votes
1answer
1k views

Why use a mixture model with RNN instead of just directly predictive real values?

Alex Graves created a model to generate hand writing sequences which use an LSTM (kind of Recurrent Neural Network) to predict the parameters for an mixture model. The mixture model is then used to ...
6
votes
1answer
3k views

Best way to initialize LSTM state

I was wondering what is the best way to initialize the state for LSTMs. Currently I just initialize it to all zeros. I can not really find anything online about how to initialize it. One thing I ...
6
votes
2answers
7k views

CNN + LSTM in tensorflow

There are quite a few examples on how to use LSTMs alone in TF, but I couldn't find any good examples on how to train CNN + LSTM jointly. From what I see, it is not quite straightforward how to do ...
6
votes
3answers
3k views

Why does the loss/accuracy fluctuate during the training? (Keras, LSTM)

I use LSTM network in Keras. During the training, the loss fluctuates a lot, and I do not understand why that would happen. Here is the NN I was using initially: And here are the loss&accuracy ...
6
votes
0answers
2k views

Which loss function to use when training LSTM for time series?

I'm experimenting with LSTM for time series prediction. The example I'm starting with uses mean squared error for training the network. I know that other time series forecasting tools use more "...
6
votes
0answers
2k views

Variable importance in RNN or LSTM

Several method have been devised for accessing or quantifying variable importance (even if only relative to each other) in MLP neural network models: Connection weights Garson’s algorithm Partial ...
6
votes
1answer
1k views

Training LSTM a sequence one item at a time

I am trying to train an lstm with a sequence and get the sequence classification for the whole sequence. I have sequences of varying length so I have one input neuron and I am feeding one item at a ...
5
votes
1answer
3k views

How to compute bits per character (BPC)?

In one of Alex Graves' papers (and several other authors as well) utilize the term bits per character (BPC). The paper that I am referencing here is "Generating Sequences with Recurrent Neural ...
5
votes
1answer
4k views

Training LSTM with and without resetting states

I'm quite new to deep learning and Keras and I want to know what is the difference between these two training methods of an LSTM RNN. ...
5
votes
2answers
4k views

Learning initial state in RNNs

I was reading Hinton's slides about recurrent networks (link), and he says that the initial state of the network should be learned just like the weights (slide 14). If that's the case, how would we ...
5
votes
1answer
2k views

How to choose between plain vanilla RNN and LSTM RNN when modelling a time series?

What are the criteria used to choose between plain vanilla RNN and LSTM RNN when you have to model a generic time series?
5
votes
1answer
2k views

How to make a LSTM to predict next word

I have made a LSTM network (hidden size 16) where I give a sequence of 10 numbers as the input and feed its output to a fully connected layer. The numbers in the sequence are corresponding to the ...
5
votes
1answer
1k views

What are 'blocks' of an LSTM?

I have read Christopher Olah's excellent LSTM article (I do not have enough reputation to post the link) and found this post and this post. Although I think I understand the mathematics and LSTMs at a ...
5
votes
1answer
380 views

Adding Features To Time Series Model LSTM

I have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to ...
5
votes
1answer
2k views

Garson's algorithm for fully connected LSTMs

Garson proposed an algorithm, later modified by Goh (1995) for determining the relative importance of an input node to a network. In the case of a single layer of hidden units, the equation is $$ Q_{...
5
votes
2answers
467 views

Does LSTM Eliminate Need for Input Lags?

Does LSTM eliminate the need for input lags? I believe the answer is yes; however, I've not found it explicitly stated in the papers and searching I have completed.
4
votes
1answer
7k views

What are “residual connections” in RNNs?

In Google's paper Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, it is stated Our LSTM RNNs have $8$ layers, with residual connections between ...
4
votes
1answer
1k views

Role of delays in LSTM networks

LSTM network is assumed to be about memory, keeping the important information for predictions. If it is the case, why do we need to consider delayed inputs as well? My assumption would be that the ...
4
votes
1answer
3k views

What's the use of the embedding matrix in a char-rnn seq2seq model?

Recently, I have been looking at seq2seq models that have been used for translating from one language to another using recurrent neural networks (often with LSTM cells). Those models can also be used ...
4
votes
1answer
834 views

Leave one out cross validation for LSTM

I have a multivariate time series data set having 6 variables. I have to predict the sixth variable at the next time step given the expected values of other five at the next time step. I am using LSTM ...