I am trying to understand the nature of LSTMs in relation to intuitions from traditional state-space models (e.g., Kalman filtering). The code below aims to simulate a simple univariate linear state-space + observation model; next, the simulated observations are set as inputs to an LSTM in sequence form with the aim of estimating the underlying state. In theory, I believe that an LSTM should be able to learn this mapping relatively easily. What I find is that this does not appear to be the case via my implementation. Feeding the training examples back into the fitted model results in state estimates that aren't what I'd consider "poor" estimates of the true states, but don't seem to be great ones either, and I'd imagine a traditional Kalman filter could do better. Am I implementing the many-to-many formulation of LSTMs incorrectly for this purpose?
For reference, I am using the LSTM implementation described in the excellent tutorial here: https://machinelearningmastery.com/timedistributed-layer-for-long-short-term-memory-networks-in-python/
import numpy as np import matplotlib.pyplot as plt from keras.models import Sequential from keras.layers import Dense, LSTM, TimeDistributed import sklearn.metrics # generate a state-space model output def genSimLinear(): N = 100 X = np.zeros((1, N, 1)) y = np.zeros((1, N, 1)) v_sigma = 1 w_sigma = 1 V = np.random.randn(X.shape, N)*v_sigma W = np.random.randn(y.shape, N)*w_sigma for it in range(1, N): X[0, it, 0] = X[0, it-1, 0] + V[0,it] y[0, it, 0] = X[0, it, 0] + W[0, it] state = X[0, 1:, 0].reshape(1, N-1, 1) measurement = y[0, 1:, 0].reshape(1, N-1, 1) return state, measurement # lstm model def model(): m = Sequential() m.add(LSTM(1, input_shape=(None, 1),\ return_sequences=True)) m.add(TimeDistributed(Dense(1))) m.compile(loss='mean_squared_error', optimizer='sgd') return m # run analysis def workflow(): # simulated raw state-space data trueState, measurement = genSimLinear() m = model() m.fit(trueState, trueState, epochs=500) stateEstimate = m.predict(measurement, batch_size=1) plt.figure(figsize=(20,20)) plt.plot(trueState[0,:,0], 'r', label='state') plt.plot(measurement[0,:,0], 'g', label='measurement') plt.plot(stateEstimate[0,:,0], 'b', label='stateEstimateLSTM') plt.grid() plt.legend() plt.title('MSE: ' +\ str(sklearn.metrics.mean_squared_error(trueState[0,:,0], xhat[0,:,0])));