Role of delays in LSTM networks

Question

LSTM network is assumed to be about memory, keeping the important information for predictions.

If it is the case, why do we need to consider delayed inputs as well?

My assumption would be that the LSTM - if the model is sufficiently complex - shall somehow remember the very last inputs if relevant. (Similar trick as if we transform a Markov Chain or higher order to a first order Markov Chain.)

However, my experiments indicate that the delayed terms matter, regardless to the complexity/simplicity of the LSTM model.

How to explain that?

Edit:

By delayed inputs, I mean the following:

$X_{t}$ is my time series. I want to predict $X_{t+1}$. I know that $X_{t+1}$ depends on $X_t$ and also on $X_{t-1}$. I would assume LSTM to be able to work even just on $X_t$.

Some code:

from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
data = [0,1,2,3,2,1]*20
import numpy as np
def shape_it(X):
    return np.expand_dims(X.reshape((-1,1)),2)

n_data = len(data)
data = np.matrix(data)
n_train = int(0.8*n_data)
X_train = shape_it(data[:,:n_train])
Y_train = shape_it(data[:,1:(n_train+1)])
X_test = shape_it(data[:,n_train:-1])
Y_test = shape_it(data[:,(n_train+1):])

model = Sequential()
model.add(SimpleRNN(units=2,activation='relu',input_shape=(None,1)))
model.add(Dense(units=5,activation='relu'))
model.add(Dense(units=1,activation='relu'))

model.compile(optimizer='adam',loss='mean_squared_error')
model.fit(X_train,Y_train.reshape(-1,1),epochs=5000,batch_size=n_train)

import matplotlib.pyplot as plt

plt.plot(model.predict(X_test).reshape(-1,1))
plt.plot(Y_test.reshape(-1,1))

Which results in the following picture:

Note that this shall be completely accurate which is obviously not that case.

An ideal answer will contain configuration of an RNN that will be capable to learn accurately, without involving $X_{t-1}$.

Answer 1

UPDATED

Your example is very interesting. On one hand it is constructed in such a way that you really need only one parameter and its value is 1: $$y_t=\beta+w y_{t-1}\\\beta=0\\w=1$$

Your training data set is small (96 observations), but with three layer network you have quite a few parameters. It's very easy to overfit.

The most interesting part is your test code. It is not clear whether you're trying to do a sequence of one-step forecasts or dynamic multi step forecast.

In one step forecast, you predict for time t and get $\hat y_t=f(x_t)=f(y_{t-1})$. So you forecast always with latest observed information to make one step ahead prediction, then proceed to the next time period.

Notice how above I'm using $y_{t-1}$ and not $\hat y_{t-1}$. That is the important distinction: in one step forecast you always use the observed value from previous step. In contrast, dynamic forecast uses the previous prediction to come up with the next: $\hat y_t=f(\hat y_{t-1})$. that is why it's called dynamic.

So, first, I re-arranged your code a little bit and modified to make it produce the one step and dynamic forecasts for comparison. Here it is with outputs followed:

# In[50]:


import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from sklearn.metrics import mean_squared_error
data = [0,1,2,3,2,1]*20
import numpy as np
def shape_it(X):
    return np.expand_dims(X.reshape((-1,1)),2)

from keras import regularizers

from numpy.random import seed



# In[51]:


n_data = len(data)
data = np.matrix(data)
n_train = int(0.8*n_data)


# In[52]:


X_train = shape_it(data[:,:n_train])
Y_train = shape_it(data[:,1:(n_train+1)])
X_test = shape_it(data[:,n_train:-1])
Y_test = shape_it(data[:,(n_train+1):])


# In[26]:


plt.plot(X_train.reshape(-1,1))
plt.plot(Y_train.reshape(-1,1))
plt.show()

# In[27]:


plt.plot(X_test.reshape(-1,1))
plt.plot(Y_test.reshape(-1,1))
plt.show()

# In[75]:


model = Sequential()
batch_size = 1
model.add(SimpleRNN(12, batch_input_shape=(batch_size, X_train.shape[1], X_train.shape[2]),stateful=True))
model.add(Dense(12))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')

epochs = 1000
for i in range(epochs):
    model.fit(X_train, np.reshape(Y_train,(-1,)), epochs=1, batch_size=batch_size, verbose=0, shuffle=False)
    model.reset_states()


# build state
model.reset_states()
model.predict(X_train, batch_size=batch_size)

predictions = list()

for i in range(len(X_test)):
    # make one-step forecast
    X = X_test[i]
    X = X.reshape(1, 1, 1)
    yhat = model.predict(X, batch_size=batch_size)[0,0]

    # store forecast
    predictions.append(yhat)
    expected = Y_test[ i ]
    print('Month=%d, Predicted=%f, Expected=%f' % (i+1, yhat, expected))

# report performance
rmse = np.sqrt(mean_squared_error(Y_test.reshape(len(Y_test)), predictions))
print('Test RMSE: %.3f' % rmse)
# line plot of observed vs predicted
plt.plot(Y_test.reshape(len(Y_test)))
plt.plot(predictions)
plt.show()

Now we've got the picture you were expecting. Your original code had a couple of issues. One is that ReLU is not a good idea for this particular problem. You have linear problem, so 'linear' or default activation should work better. The second issue is that you have to call with stateful=True in fit function. Finally, I changed the prediction implementation to make it one step forecast.

This is not bad, but it's only one step forecast. Next we'll try to do the dynamic forecast as explained earlier.

# build state
model.reset_states()
model.predict(X_train, batch_size=batch_size)

dynpredictions = list()
dyhat = X_test[0]


for i in range(len(X_test)):
    # make one-step forecast
    dyhat = yhat.reshape(1, 1, 1)
    dyhat = model.predict(dyhat, batch_size=batch_size)[0,0]

    # store forecast
    dynpredictions.append(dyhat)
    expected = Y_test[ i ]
    print('Month=%d, Predicted Dynamically=%f, Expected=%f' % (i+1, dyhat, expected))


drmse = np.sqrt(mean_squared_error(Y_test.reshape(len(Y_test)), dynpredictions))
print('Test Dynamic RMSE: %.3f' % drmse)
# line plot of observed vs predicted
plt.plot(Y_test.reshape(len(Y_test)))
plt.plot(dynpredictions)
plt.show()

The dynamic forecast doesn't look so good as seen below. Recall that now we're out of sample, and we are not using observed values beyond observation #96 unlike in one step forecast. Still, we want to nail it, because the problem is so obvious to us that we want NN to figure it too.

I'm going to try a different NN with just one hidden layer, and regularization to fight overfitting as follows.

seed(1)

modelR = Sequential()
batch_size = 1
modelR.add(SimpleRNN(4, batch_input_shape=(batch_size, X_train.shape[1], X_train.shape[2]),stateful=True,
                     kernel_regularizer=regularizers.l2(0.01),
                activity_regularizer=regularizers.l1(0.)))
modelR.add(Dense(1,kernel_regularizer=regularizers.l2(0.01),
                activity_regularizer=regularizers.l1(0.)))
modelR.compile(loss='mean_squared_error', optimizer='adam')

epochs = 1000
for i in range(epochs):
    modelR.fit(X_train, np.reshape(Y_train,(-1,)), epochs=1, batch_size=batch_size, verbose=0, shuffle=False)
    modelR.reset_states()

# build state
modelR.reset_states()
modelR.predict(X_train, batch_size=batch_size)

predictions = list()

for i in range(len(X_test)):
    # make one-step forecast
    X = X_test[i]
    X = X.reshape(1, 1, 1)
    yhat = modelR.predict(X, batch_size=batch_size)[0,0]

    # store forecast
    predictions.append(yhat)
    expected = Y_test[ i ]
    print('Month=%d, Predicted=%f, Expected=%f' % (i+1, yhat, expected))

# report performance
rmse = np.sqrt(mean_squared_error(Y_test.reshape(len(Y_test)), predictions))
print('Test RMSE: %.3f' % rmse)
# line plot of observed vs predicted
plt.plot(Y_test.reshape(len(Y_test)))
plt.plot(predictions)
plt.show()

The new model is still working on the one step forecast, as seen below.

Let's try the dynamic forecast now.

# build state
modelR.reset_states()
modelR.predict(X_train, batch_size=batch_size)

dynpredictions = list()
dyhat = X_test[0]


for i in range(len(X_test)):
    # make one-step forecast
    dyhat = dyhat.reshape(1, 1, 1)
    dyhat = modelR.predict(dyhat, batch_size=batch_size)[0,0]

    # store forecast
    dynpredictions.append(dyhat)
    expected = Y_test[ i ]
    print('Month=%d, Predicted Dynamically=%f, Expected=%f' % (i+1, dyhat, expected))


drmse = np.sqrt(mean_squared_error(Y_test.reshape(len(Y_test)), dynpredictions))
print('Test Dynamic RMSE: %.3f' % drmse)
# line plot of observed vs predicted
plt.plot(Y_test.reshape(len(Y_test)))
plt.plot(dynpredictions)
plt.show()

Now the dynamic forecast seems to be working too!