I understand that similar questions have been asked before, but they are all based on specific examples. I want to consider a very simple example: we have a sequence of 1000 numbers, and want an LSTM to predict the average of the last three numbers for each number. So:
[-, -, 1.67, 2.33, 3.67, 4.667,...]
We could 'pad' the first values by averaging backwards as much as we can:
[0, 0.5, 1.67, 2.33, 3.67, 4.667,...]
Via numpy, I create this dataset as follows:
input = np.random.randint(0, 10, size=(1000,)) # Output is average of i-2, i-1 and i output = ; output.append(input) # does not work for i = 0; output.append((input + input) / 2) # does not work for i = 1 for i in range(2, len(input)): output.append((input[i-2] + input[i-1] + input[i]) / 3) # for all i > 1 output = np.asarray(output)
Now I would like to train an LSTM-based network to this. I create the network as follows:
model = Sequential() model.add(LSTM(4, input_shape=(1, 1))) model.add(Dense(1, activation='linear')) model.compile(loss='mse', optimizer=keras.optimizers.Adam(learning_rate=0.001))
I reshape the input and train the model:
input = np.reshape(input, (input.shape, 1, 1)) # based on similar questions model.fit(input, output, epochs=20, batch_size=1, verbose=1)
Now, the mean square error does not decrease far below 2. Even if I add way more neurons to the LSTM layer, there is no performance increase. It seems like I have mis-shaped my input.
What should the shape of my input be?
PS: normalizing the input / outputs does not help.