Proper shape of LSTM dataset for keras

Question

I understand that similar questions have been asked before, but they are all based on specific examples. I want to consider a very simple example: we have a sequence of 1000 numbers, and want an LSTM to predict the average of the last three numbers for each number. So:

[0,1,4,2,5,7,...] -> [-, -, 1.67, 2.33, 3.67, 4.667,...]

We could 'pad' the first values by averaging backwards as much as we can:

[0,1,4,2,5,7,...] -> [0, 0.5, 1.67, 2.33, 3.67, 4.667,...]

Via numpy, I create this dataset as follows:

input = np.random.randint(0, 10, size=(1000,))

# Output is average of i-2, i-1 and i
output = [];
output.append(input[0]) # does not work for i = 0;
output.append((input[0] + input[1]) / 2) # does not work for i = 1

for i in range(2, len(input)):
    output.append((input[i-2] + input[i-1] + input[i]) / 3) # for all i > 1

output = np.asarray(output)

Now I would like to train an LSTM-based network to this. I create the network as follows:

model = Sequential()
model.add(LSTM(4, input_shape=(1, 1)))
model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer=keras.optimizers.Adam(learning_rate=0.001))

I reshape the input and train the model:

input = np.reshape(input, (input.shape[0], 1, 1)) # based on similar questions
model.fit(input, output, epochs=20, batch_size=1, verbose=1)

Now, the mean square error does not decrease far below 2. Even if I add way more neurons to the LSTM layer, there is no performance increase. It seems like I have mis-shaped my input.

What should the shape of my input be?

PS: normalizing the input / outputs does not help.

Answer 1

So @wprime gave a part of the answer. Indeed, we want to set return_sequences=True because we don't just want the final prediction for each sequence, we want all the predictions along the way as well. By then reshaping the input correctly ([batches, timesteps, features]) we get a very good result. This is minimal working example:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
import keras
import numpy as np

# Generate dataset
input = np.random.randint(0, 10, size=(2000,))

# Output is average of i-2, i-1 and i
output = [];
output.append(input[0]) # does not work for i = 0;
output.append((input[0] + input[1]) / 2) # does not work for i = 1

for i in range(2, len(input)):
    output.append((input[i-2] + input[i-1] + input[i]) / 3) # for all i > 1

output = np.asarray(output);

# Normalize to range 0-1
input = input / 10;
output = output / 10;

# Split into train/valid with 50/50 ratio
split = int(input.shape[0] * 0.5)
x_train = input[:split];
x_valid = input[split:];

y_train = output[:split];
y_valid = output[split:];

# Reshape input into [batch, timesteps, features]
x_train = np.reshape(x_train, (1,x_train.shape[0], 1))
x_valid = np.reshape(x_valid, (1,x_valid.shape[0], 1))

y_train = np.reshape(y_train, (1,y_train.shape[0], 1))
y_valid = np.reshape(y_valid, (1,y_valid.shape[0], 1))

# Create and train LSTM network
model = Sequential()
model.add(LSTM(4, input_shape=(1000, 1),return_sequences=True))
model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer=keras.optimizers.Adam(learning_rate=0.1))
model.fit(x_train, y_train, validation_data=(x_valid, y_valid), epochs=400, batch_size=1, verbose=2)

# Validate the results
predictions = np.squeeze(model.predict(x_valid)) * 10;
y_valid = np.squeeze(y_valid) * 10
print(predictions[50:55])
print(y_valid[50:55])
```

Answer 2

The shape of LSTM inputs needs to be a 3D tensor of dimensions, [batch_size, time_steps, num_features]. Your response variable can either be a separate Numpy array of shape [batch_size, time_steps, 1] or it can be included with the feature; you just need to split it off when passing in training data. It is worth noting, batch size defaults to 32 in Keras/TF.

On the topic of changing the number of nodes in the LSTM layer, I don't see why it would help. Passing in a single observation of a single variable will have the same effect on each node, each node would not weigh the same input differently making the number of them irrelevant. When you BPTT each node would affect the loss the same as they are all replicates. Think of the acyclic graph of layers here, the input gets sent to the same four LSTM nodes, and then condensed back into a single node.

I don't think it's a shaping error but may have something to do with the "return_sequences" parameter of the LSTM layer. Look further into the functionality of this, and see if you are using it correctly, it defaults to "False".

TensorFlow has very useful blog post titled "Time series forecasting" which reviews data preprocessing for LSTM networks, and the LSTM documentation is also useful.