Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [lstm]

A Long Short Term Memory (LSTM) is a neural network architecture that contains recurrent NN blocks that can remember a value for an arbitrary length of time.

30
votes
3answers
14k views

How does LSTM prevent the vanishing gradient problem?

LSTM was invented specifically to avoid the vanishing gradient problem. It is supposed to do that with the Constant Error Carousel (CEC), which on the diagram below (from Greff et al.) correspond to ...
9
votes
3answers
8k views

Structure of Recurrent Neural Network (LSTM, GRU)

I am trying to understand the architecture of RNNs. I have found this tutorial which has been very helpful: http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Especially this image: How ...
1
vote
1answer
506 views

Deep learning model (LSTM) with temporal and non temporal attributes

I'm working on a project to predict the usage of all the files(rough frequency of usage) in a filesystem (a company server on which 100s of company employees are active) in near future (say the next 1 ...
28
votes
4answers
17k views

Understanding LSTM units vs. cells

I have been studying LSTMs for a while. I understand at a high level how everything works. However, going to implement them using Tensorflow I've noticed that BasicLSTMCell requires a number of units (...
19
votes
3answers
9k views

What are the advantages of stacking multiple LSTMs?

What are the advantages, why would one use multiple LSTMs, stacked one side-by-side, in a deep-network? I am using a LSTM to represent a sequence of inputs as a single input. So once I have that ...
8
votes
2answers
10k views

What is the output of a tf.nn.dynamic_rnn()?

I am not sure about what I understand from the official documentation, which says: Returns: A pair (outputs, state) where: outputs: The RNN output Tensor. ...
3
votes
1answer
689 views

Sales forecast in presence of stockout

Can anyone provide some idea about how to model sales with stockout? (it could be a general answer or specific to any modeling methods) If I want to model this with LSTM with multiple time steps, how ...
4
votes
1answer
337 views

State-of-the-art algorithms for the training of neural networks with GRU or LSTM units

I recently read a lot about neural networks using GRU or LSTM units. There are many easy to use frameworks like tensorflow that do not even require high knowledge about programming. Unfortunately, I ...
4
votes
1answer
2k views

Is anyone stacking LSTM and GRU cells together and why?

TensorFlow allows you to create MultiRNNCell composed sequentially of multiple simple cells (LSTM and GRU). I usually use same type of cell when creating MultiRNNCell but I was wondering if there ...
4
votes
1answer
832 views

Leave one out cross validation for LSTM

I have a multivariate time series data set having 6 variables. I have to predict the sixth variable at the next time step given the expected values of other five at the next time step. I am using LSTM ...
1
vote
1answer
2k views

Time steps in Keras LSTM

My understanding of time-series LSTM training is that the recurrent cell gets unrolled to a specified length (num_steps), and parameter updates are back-propagated ...
19
votes
3answers
34k views

Using RNN (LSTM) for predicting the timeseries vectors (Theano)

I have very simple problem but I cannot find a right tool to solve it. I have some sequence of vectors of the same length. Now I would like to train LSTM RNN on train sample of these sequences and ...
15
votes
1answer
15k views

Difference between samples, time steps and features in neural network

I am going through the following blog on LSTM neural network: http://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/ The author reshapes the input ...
14
votes
1answer
15k views

What is a feasible sequence length for an RNN to model?

I'm looking into using a LSTM (long short-term memory) version of a recurrent neural network (RNN) for modeling timeseries data. As the sequence length of the data increases, the complexity of the ...
23
votes
1answer
36k views

Training loss goes down and up again. What is happening?

My training loss goes down and then up again. It is very weird. The cross-validation loss tracks the training loss. What is going on? I have two stacked LSTMS as follows (on Keras): ...
21
votes
1answer
2k views

What are attention mechanisms exactly?

Attention mechanisms have been used in various Deep Learning papers in the last few years. Ilya Sutskever, head of research at Open AI, has enthusiastically praised them: https://towardsdatascience....
11
votes
2answers
2k views

Why can RNNs with LSTM units also suffer from “exploding gradients”?

I have a basic knowledge of how RNNs (and, in particular, with LSTMs units) work. I have a pictorial idea of the architecture of an LSTM unit, that is a cell and a few gates, which regulate the flow ...
4
votes
2answers
4k views

How can calculate number of weights in LSTM

If I have a LSTM with below parameters, how can I calculate the total number of wights ? Input 39 Output 34 Hidden Layers = 3 Cells in each layer = 1024 I saw ...
5
votes
2answers
4k views

Learning initial state in RNNs

I was reading Hinton's slides about recurrent networks (link), and he says that the initial state of the network should be learned just like the weights (slide 14). If that's the case, how would we ...
3
votes
0answers
291 views

Truncated Back Propagation of LSTM with K2 > K1

Every K1 time steps we will run truncated back prop through time (BPTT) for k2 time steps. For k2 <= k1 I don't find any problems. But let's look at the case where k2 > k1, and particularly where ...
3
votes
1answer
155 views

Cannot understand LSTM inference

I seem to have stumbled on a hole in my understanding around LSTMs. In short, I cannot understand how even a simple one is actually fed samples, upon inference time/training time. Here are the details:...
3
votes
0answers
717 views

Why is the derivative of the LSTM cell state w.r.t. to the previous cell state equal to the forget gate?

I keep seeing this online, on Quora and Machine Learning subreddits but I don't get it. Here's some basic math to show otherwise: We use this equation for the cell state: $c_t = f_t \odot c_t\__1 + ...
2
votes
2answers
91 views

How does an LSTM process sequences longer than its memory?

* Note: The premise of my question was incorrect in the first place. My question assumes that an LSTM maintains a separate set of weights for each time step in the memory it is given as a design ...
2
votes
1answer
733 views

How to design architecture for LSTM neural network

Is there some guideline to designing architecture for neural networks? I want to use LSTM network for predicting in time series. I have a small dictionary variation (8 values) but lot of their ...
1
vote
0answers
157 views

Adding noise to time series data to increase training data

I am dealing with a weekly time series forecasting problem and I am currently investigating the use of an LSTM to make a multi-step forecast for a univariate time series. I actually have a ...
1
vote
1answer
798 views

Back propagation in seq2seq models

I've implemented a seq2seq model for character-based text mirroring as a part of Udacity's Deep Learning class (here's the code). My model is very basic because it's a single LSTM as both encoder and ...
1
vote
1answer
198 views

LSTM Classifying all the words as the same class

I've used Lasagne to build a LSTM model to classify words with the IOB-tags. About 25-40% of the training words classes is O, thus receiving the same int32 class number 126. The words go through a ...
1
vote
0answers
705 views

3d Convolution vs CNN-LSTM for Gesture recognition

I want to implement a gesture recognition system from video (of hand movements). Some people have experimented with 3d convolutions to extract not only spatial features out of images, but also ...
0
votes
1answer
91 views

Is it correct to evaluate Neural Network after a fixed number of batch-updates, rather than at the end of epoch?

I'm training a neural network on a number of datasets of different size with a fixed batch size and an exponential learning decay. Normally, I would evaluate model performance, save checkpoint and ...