Skip to main content
59 votes
Accepted

Difference between feedback RNN and LSTM/GRU

All RNNs have feedback loops in the recurrent layer. This lets them maintain information in 'memory' over time. But, it can be difficult to train standard RNNs to solve problems that require learning ...
user20160's user avatar
  • 32.9k
43 votes
Accepted

Why are the weights of RNN/LSTM networks shared across time?

The accepted answer focuses on the practical side of the question: it would require a lot of resources, if there parameters are not shared. However, the decision to share parameters in an RNN has been ...
Maxim's user avatar
  • 3,319
43 votes
Accepted

Understanding input_shape parameter in LSTM with Keras

LSTM shapes are tough so don't feel bad, I had to spend a couple days battling them myself: If you will be feeding data 1 character at a time your input shape should be (31,1) since your input has 31 ...
tRosenflanz's user avatar
40 votes
Accepted

Is LSTM (Long Short-Term Memory) dead?

Maybe. But RNNs aren't. Transformers learn "pseudo-temporal" relationships; they lack the true recurrent gradient that RNNs have, and thus extract fundamentally different features. This ...
OverLordGoldDragon's user avatar
39 votes
Accepted

RNNs: When to apply BPTT and/or update weights?

I'll assume we're talking about recurrent neural nets (RNNs) that produce an output at every time step (if output is only available at the end of the sequence, it only makes sense to run backprop at ...
user20160's user avatar
  • 32.9k
35 votes
Accepted

How does LSTM prevent the vanishing gradient problem?

The vanishing gradient is best explained in the one-dimensional case. The multi-dimensional is more complicated but essentially analogous. You can review it in this excellent paper [1]. Assume we ...
bayerj's user avatar
  • 13.8k
35 votes

What are "residual connections" in RNNs?

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear ...
Hugh Perkins's user avatar
  • 4,797
31 votes

Is LSTM (Long Short-Term Memory) dead?

It is funny that you ask now, since just today I came across a paper by Wang, Khabsa, and Ma (2020) To Pretrain or Not to Pretrain who show that if you have large enough training set, the difference ...
Tim's user avatar
  • 140k
30 votes
Accepted

Understanding LSTM units vs. cells

The terminology is unfortunately inconsistent. num_units in TensorFlow is the number of hidden states, i.e. the dimension of $h_t$ in the equations you gave. Also, ...
Franck Dernoncourt's user avatar
30 votes
Accepted

What are attention mechanisms exactly?

Attention is a method for aggregating a set of vectors $v_i$ into just one vector, often via a lookup vector $u$. Usually, $v_i$ is either the inputs to the model or the hidden states of previous time-...
shimao's user avatar
  • 26.4k
25 votes

How does LSTM prevent the vanishing gradient problem?

I'd like to add some detail to the accepted answer, because I think it's a bit more nuanced and the nuance may not be obvious to someone first learning about RNNs. For the vanilla RNN, $$\frac{\...
Kevin's user avatar
  • 401
25 votes
Accepted

Difference between samples, time steps and features in neural network

I found this just below the [samples, time_steps, features] you are concerned with. X = numpy.reshape(dataX, (len(dataX), seq_length, 1)) Samples - This is the ...
Joonatan Samuel's user avatar
22 votes
Accepted

How to train LSTM model on multiple time series data?

Make the identity of the agent one of the features, and train on all data. Probably train on a mini-batch of eg 128 agents at a time: run through the time-series from start to finish for those 128 ...
Hugh Perkins's user avatar
  • 4,797
17 votes
Accepted

How to compute bits per character (BPC)?

From my understanding, the BPC is just the average cross-entropy (used with log base 2). In the case of Alex Graves' papers, the aim of the model is to approximate the probability distribution of the ...
user6952886's user avatar
17 votes
Accepted

Activation function between LSTM layers

The purpose of the Rectified Linear Activation Function (or ReLU for short) is to allow the neural network to learn nonlinear dependencies. Specifically, the way this works is that ReLU will return ...
Michael Grogan's user avatar
17 votes

Forecasting Prices vs Returns by Deep Learning

What you've outlined is probably the single most common error that machine learning researchers make when analyzing financial data: it's trivial to discover that a great predictor of tomorrow's price ...
Sycorax's user avatar
  • 92.5k
16 votes

Why are the weights of RNN/LSTM networks shared across time?

The 'shared weights' perspective comes from thinking about RNNs as feedforward networks unrolled across time. If the weights were different at each moment in time, this would just be a feedforward ...
user20160's user avatar
  • 32.9k
16 votes

Difference between feedback RNN and LSTM/GRU

Standard RNNs (Recurrent Neural Networks) suffer from vanishing and exploding gradient problems. LSTMs (Long Short Term Memory) deal with these problems by introducing new gates, such as input and ...
user139688's user avatar
16 votes

Why does the loss/accuracy fluctuate during the training? (Keras, LSTM)

There are several reasons that can cause fluctuations in training loss over epochs. The main one though is the fact that almost all neural nets are trained with different forms of stochastic gradient ...
SaTa's user avatar
  • 361
15 votes

What optimization methods work best for LSTMs?

Ironically the best Optimizers for LSTMs are themselves LSTMs: https://arxiv.org/abs/1606.04474 Learning to learn by gradient descent by gradient descent. The basic idea is to use a neural network (...
Anona112's user avatar
  • 259
15 votes

Is LSTM (Long Short-Term Memory) dead?

Our group recently built an LSTM model in a real world application. At first we had used other approaches, but then we decided to include features that were measurements taken over time, but of ...
rumtscho's user avatar
  • 1,829
14 votes
Accepted

Variable importance in RNN or LSTM

In short, yes, you can get some measure of variable importances for RNN based models. I won't iterate through all of the listed suggestions in the question, but I will walk through an example of ...
kbrose's user avatar
  • 1,427
14 votes

Understanding LSTM units vs. cells

Most LSTM/RNN diagrams just show the hidden cells but never the units of those cells. Hence, the confusion. Each hidden layer has hidden cells, as many as the number of time steps. And further, each ...
Garima Jain's user avatar
14 votes

Understanding input_shape parameter in LSTM with Keras

Check this git repository LSTM Keras summary diagram and i believe you should get everything crystal clear. This git repo includes a Keras LSTM summary diagram that shows: the use of parameters like ...
Mohammad Fneish's user avatar
14 votes

Why can RNNs with LSTM units also suffer from "exploding gradients"?

A very short answer: LSTM decouples cell state (typically denoted by c) and hidden layer/output (typically denoted by h), and ...
soloice's user avatar
  • 242
13 votes

Preventing overfitting of LSTM on small dataset

You could try: Reduce the number of hidden units, I know you said it already seems low, but given that the input layer only has 80 features, it actually can be that 128 is too much. A rule of thumb ...
Miguel's user avatar
  • 1,436
13 votes

What are the advantages of stacking multiple LSTMs?

From {1}: While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that deep RNNs work better than shallower ones on some ...
Franck Dernoncourt's user avatar
13 votes

Best way to initialize LSTM state

Normally, you would set the initial states to zero, but the network is going to learn to adapt to that initial state. The following article suggests learning the initial hidden states or using random ...
jpeg729's user avatar
  • 131
13 votes
Accepted

If we primarily use LSTMs over RNNs to solve the vanishing gradient problem, why can't we just use ReLUs/leaky ReLUs with RNNs instead?

I think there's some confusion here. The reason you have vanishing gradients in neural networks (with say, softmax) is wholly different from RNNs. With neural networks, you get vanishing gradients ...
Alex R.'s user avatar
  • 14k
12 votes

How to make LSTM predict multiple time steps ahead?

There are different approaches Recursive strategy one many-to-one model ...
mingxue's user avatar
  • 221

Only top scored, non community-wiki answers of a minimum length are eligible