# Training LSTM with and without resetting states

I'm quite new to deep learning and Keras and I want to know what is the difference between these two training methods of an LSTM RNN.

1:
for i in range(10): #training
model.fit(trainX, trainY, epochs=1, batch_size=batch_size, verbose=0,
shuffle=False)
model.reset_states()

2:
model.fit(trainX, trainY, epochs=10, batch_size=batch_size, verbose=0,
shuffle=False)


In both cases, doesn't the network train 10 times over the whole dataset? I realize that in example one we can reset the state in each data batch iteration but even if I delete the reset instruction the results are quite different. I'm confused.

• Sorry by the tabulation, was my first post on the forum. Thanks! – J.Cirera Jul 26 '17 at 10:53

Yes, you are right. In both cases, the model is trained for 10 epochs. During each epoch, all examples in your training data flow through the network. The batch size determines the number of examples after which the weights or parameters of the model are updated.

The difference between the first and the second case is that the first one allows you to perform some processing outside the fit() method between the epochs, such as model.reset_states(). However, similar processing can also be applied to the second case within the fit() method via custom callbacks class, which include, for example, on_epoch_begin, on_epoch_end, on_batch_begin and on_batch_end functions.

Regarding the problem of getting quite different results with the two cases when model.reset_states() is removed from the first one: it shouldn't happen. You will get different results from each case if you reset the states of the model between the epochs in one case but not in the other. The results (loss after a certain number of epochs) will be the same if you don't reset the states in either case between the epochs, initialize a pseudorandom number generator before importing Keras and restart the Python interpreter between running the two cases. I validated this with the following example, where the objective is to learn a pure sine wave from a noisy one. The following code snippet has been implemented with Python 3.5, NumPy 1.12.1, Keras 2.0.4 and Matplotlib 2.0.2.:

import numpy as np

# Needed for reproducible results
np.random.seed(1)

from keras.models import Sequential
from keras.layers import LSTM, Dense

# Generate example data
# -----------------------------------------------------------------------------
x_train = y_train = [np.sin(i) for i in np.arange(start=0, stop=10, step=0.01)]
noise = np.random.normal(loc=0, scale=0.1, size=len(x_train))
x_train += noise

n_examples = len(x_train)
n_features = 1
n_outputs = 1
time_steps = 1

x_train = np.reshape(x_train, (n_examples, time_steps, n_features))
y_train = np.reshape(y_train, (n_examples, n_outputs))

# Initialize LSTM
# -----------------------------------------------------------------------------
batch_size = 100
model = Sequential()
return_sequences=True, stateful=True, batch_size=batch_size))

# Train LSTM
# -----------------------------------------------------------------------------
epochs = 70

# Case 1
for i in range(epochs):
model.fit(x_train, y_train, epochs=1, batch_size=batch_size, verbose=2,
shuffle=False)

# !!! To get exactly the same results between the cases, do the following:
# !!!  * To record the loss of the 1st case, run all the code until here.
# !!!  * To record the loss of the 2nd case,
# !!!    restart Python, comment out the 1st case and run all the code.

# Case 2
model.fit(x_train, y_train, epochs=epochs, batch_size=batch_size, verbose=2,
shuffle=False)


As an extra, here is a visualization of the results of either case where states weren't reset:

import matplotlib.pyplot as plt

plt.style.use('ggplot')
ax.plot(x_train[:, 0], label='x_train', color='#111111', alpha=0.8, lw=3)
ax.plot(y_train[:, 0], label='y_train', color='#E69F00', alpha=1, lw=3)
ax.plot(model.predict(x_train, batch_size=batch_size)[:, 0],
label='Predictions for x_train after %i epochs' % epochs,
color='#56B4E9', alpha=0.8, lw=3)
plt.legend(loc='lower right')


On the Keras website, the statefulness of RNNs is discussed in recurrent layers documentation and in FAQ.

Edit: The above solution currently works with Theano backend but not with TensorFlow backend.

• Do recall what backend you were using for Keras? I'm getting significantly worse results when running that same code using TensorFlow with Python 3.5 and somewhat newer versions of Keras and NumPy. So, I'm wondering if I've picked up something defective. – Josh Heitzman Feb 19 '19 at 5:55
• @JoshHeitzman I must have been using Theano because there is currently an open issue on Keras' GitHub repo regarding reproducibility with TensorFlow. I have edited my answer to include a link to the issue. Thanks for the heads up! – tuomastik Feb 19 '19 at 6:05
• Thanks! I switched my backend to the current version of Theano and now get a good fit that resembles yours, instead of the very bad fit I observed from TensorFlow. – Josh Heitzman Feb 20 '19 at 4:34