Trouble training LSTM for sequence to sequence learning of sensor time series

Question

I'm experimenting with using RNNs/LSTMs in place of a Kalman Filter (KF) for sensor fusion. I'm struggling to make much progress, and would appreciate some feedback/advice.

I have several multi-variate time series of multi-model sensor data. I also have an equal length multi-variate time series that I'd like to predict given the sensor input that will be my target for training. Specifically for my experiments the target will be the output states of a non-linear Kalman filter for each time step applied to the input data.

A KF takes measurements and an internal state as input and produces a new internal state and a prediction at each time step. At high level an RNN does the same, so it would seem to me that a RNN should be able to reproduce the results of a KF pretty accurately.

I've been experimenting with sequence-to-sequence models where my input data are shaped (N_batch, N_samples, N_dim_in) and my output data are (N_batch, N_samples, N_dim_out). I have a lot of data, with hundreds of thousands of N_sample windows to use for training.

• I've been using fairly short sample windows, with N_samples=10.
• I've been training in such a way that I carry the LSTM state from one batch to the next, using the end state from batch i as the starting state for batch i+1.
• I use a tanh activation on the hidden layers, and a linear activation on the output layer.
• I'm using Mean Squared Error (MSE) as my loss function.
• I have scaled my input and targets to the range [0, 1].
• I've been using a batch size of 32.
• For regularization, I have mostly been using dropout if at all.

For reference, here is the Keras code that I use to build my models:

def get_model(layers, dropouts):
    """LAYERS is a list of number of nodes per hidden layer. DROPOUTS
    is an equal length list of dropout fractions for dropout layers to
    include after each hidden layer (None or 0 for no dropout)."""

    model = Sequential()
    for i, (layer, dropout) in enumerate(zip(layers, dropouts)):
        kwargs = {
            'return_sequences': True,
            'stateful': True,
            'implementation': 2,
            'activation': 'tanh'
        }

        if i == 0:
            kwargs['batch_input_shape'] = (32, 10, 7)

        model.add(
            LSTM(layer, **kwargs)
        )
        if dropout is not None and dropout > 0.0:
            model.add(Dropout(dropout))

    model.add(
        LSTM(
            3, 
            return_sequences=True, 
            implementation=2,
            stateful=True
        )
    )

    return model

# Example: To create a model with a 64 node and a 32 node hidden layer
# with a 50% dropout layer between the second hidden and output layers
# There is always a 6 node output layer with linear activation
model = get_model([64, 32], [None, 0.5])
opt = RMSprop()
model.compile(loss='mean_squared_error', optimizer=opt)
model.fit(
    sX, sY, # Scaled input and output
    validation_data=(sXval, sYval),  # scaled validation input and output
    batch_size=32,
    initial_epoch=0,
    epochs=50,
    shuffle=False
)

I've been struggling to get anywhere near my target loss for the problem at hand, or getting the model to make much progress toward learning. At the moment, I'm just trying to get the model to overfit without worrying about generalization to validation sets. I keep hitting a wall in MSE measured on training data.

I've experimented with many different widths and depths of the network, different learning rates, and dropout regularization. I would expect that I would be able to add nodes/layers and see improvement, or train for more epochs with more data, but I either stall out at an error far above my target or performance gets worse. And I'm not seeing any improvement in validation loss with more dropout.

I'm still convinced this should work, but there are so many things to try at this point that I'm at a bit of a loss as to where to focus my attention. I've been looking through the literature to see if there are examples of people doing what I'm trying to do here but haven't had much luck. In the paper https://arxiv.org/abs/1701.08376, which is one of the most analogous examples I've come across, they state:

Furthermore, in the traditional LSTM model, the hidden state is carried over to the next time-step, but the output itself is not fed back to the input. In the case of odometry estimation the availability of the previous state is particularly important as the output is essentially an accumulation of incremental displacements at each step. Thus for our model, we directly connect the output pose produced by the SE(3) concatenation layer, back as input to the Core LSTM for the next timestep.

but it's not clear to me that a) this is necessary in my case, or b) how to do such a thing at all. It's non-trivial to implement it seems, so I don't want to go down that path unless there is a good reason.

Here are some specific questions I've been pondering:

• Is there some fundamental flaw in my approach? A reason why a sequence-to-sequence LSTM network wouldn't be able to replicate a non-linear Kalman filter?
• Why am I getting worse training accuracy with more layers/nodes per layer?
  • Could I be running into issues with low gradients preventing backprop from flowing errors through the network? Could some form of regularization, maybe e.g. batch norm, overcome the issue?
  • Or do I need to reduce my learning rate as the network grows?
• Do I need longer time windows? Shorter?
  • A KF only depends on information from the previous time step, so I might actually think that N_samples=2 would be sufficient, but I'm not sure.
• Is the stateful LSTM model the right approach here?
• Does anyone know of any references/examples that might be useful?

Any advice or feedback would be greatly appreciated. Thanks!

Answer 1

I am working on a similar project. I'll try to answer the questions based on my understanding.

1- I think your approach is basically correct.

2- The reason why you are getting worse is not that much clear. What I would suggest to do change stateful=True into False.

The stateful LSTM is really not easy to understand. If your sequence is not that much long, say less than 200 time steps, the stateless LSTM is enough to deal with it. What you need to do is feed the data in shape (batch_size, number_time_step, dimension_in_size) into the stateless LSTM. Note here by one time step in the sequence, I mean one sample sensor data point. Given the example, you only consider 10 time steps and dimension is 7, I think one layer should give reasonable result.