I am trying to understand the nature of LSTMs in relation to intuitions from traditional state-space models (e.g., Kalman filtering). The code below aims to simulate a simple univariate linear state-space + observation model; next, the simulated observations are set as inputs to an LSTM in sequence form with the aim of estimating the underlying state. In theory, I believe that an LSTM should be able to learn this mapping relatively easily. What I find is that this does not appear to be the case via my implementation. Feeding the training examples back into the fitted model results in state estimates that aren't what I'd consider "poor" estimates of the true states, but don't seem to be great ones either, and I'd imagine a traditional Kalman filter could do better. Am I implementing the many-to-many formulation of LSTMs incorrectly for this purpose?

For reference, I am using the LSTM implementation described in the excellent tutorial here: https://machinelearningmastery.com/timedistributed-layer-for-long-short-term-memory-networks-in-python/

import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense, LSTM, TimeDistributed
import sklearn.metrics

# generate a state-space model output
def genSimLinear():
    N = 100
    X = np.zeros((1, N, 1))
    y = np.zeros((1, N, 1))
    v_sigma = 1
    w_sigma = 1
    V = np.random.randn(X.shape[0], N)*v_sigma
    W = np.random.randn(y.shape[0], N)*w_sigma
    for it in range(1, N):
        X[0, it, 0] = X[0, it-1, 0] + V[0,it]
        y[0, it, 0] = X[0, it, 0] + W[0, it]
    state = X[0, 1:, 0].reshape(1, N-1, 1)
    measurement = y[0, 1:, 0].reshape(1, N-1, 1)
    return state, measurement

# lstm model
def model():
    m = Sequential()
    m.add(LSTM(1, input_shape=(None, 1),\
    m.compile(loss='mean_squared_error', optimizer='sgd')
    return m

# run analysis
def workflow():
    # simulated raw state-space data
    trueState, measurement = genSimLinear()

    m = model()
    m.fit(trueState, trueState, epochs=500)
    stateEstimate = m.predict(measurement, batch_size=1)

    plt.plot(trueState[0,:,0], 'r', label='state')
    plt.plot(measurement[0,:,0], 'g', label='measurement')
    plt.plot(stateEstimate[0,:,0], 'b', label='stateEstimateLSTM')
    plt.title('MSE: ' +\
              str(sklearn.metrics.mean_squared_error(trueState[0,:,0], xhat[0,:,0])));

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.