Why is my LSTM +- 1DConvNet so ineffective at waveform analysis?

Question

I'm trying to learn LSTMs and I thought a nice way of doing it would be identifying onset-and-offset of QRS complexes on ECGs.

I have 300 x 200 x 2 numpy array of ECGs (300 ECGs, each of 200 data points, each data point being [x=time,y=voltage]. Sometimes it's 0 padded at the start (like the example below) as ECGs can be of different durations.

I also have labels in the form 300 x 2, 300 labels of [qrs_onset,qrs_offset].

This is an example of an ECG plotted and labels superimposed.

I've done some pre-processing, shuffling the location of the ECG forwards and backwards by ~ 10% of the cycle length. I then feed it into an LSTM model with Python/Keras/Tensorflow backend:

self.model = Sequential()
self.model.add(LSTM(input_length=199, input_dim=2, dropout=0.1, output_dim=64, return_sequences=True))
self.model.add(LSTM(256, dropout=0.1, return_sequences=True))
self.model.add(LSTM(100, dropout=0.1, return_sequences=False))
self.model.add(Dense(2))
self.model.add(Activation("linear"))
self.model.compile(loss="mean_squared_error", optimizer="rmsprop")

However, all the LSTM does is fine a location that fits the entire dataset best, and gives that exact location regardless of the ECG fed to it.

It takes about 4000 epochs to get to this point, but at this stage the loss completely plateaus and makes 0 further progress.

It's strange because I thought an LSTM would be great for this task.

Things I have tried:

1. Reducing the epoch number to a smaller proportion of the dataset to get out of local minima
2. Adding between 1 and 3 LSTM layers of between 64 and 300 units.
3. Using the Adam optimiser instead of RMSProp
4. Adding and removing dropout between 0.0 and 0.2

5. Adding a 1D Conv layer to try and identify the peaks using spatial information before feeding this into an LSTM

   self.model = Sequential()
   self.model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))
   self.model.add(MaxPooling1D(pool_size=2))
   self.model.add(
       LSTM(300, dropout=0.2, return_sequences=True, input_shape=(None, 300)))
   self.model.add(
       LSTM(300, dropout=0.2, return_sequences=True, input_shape=(None, 300)))
   self.model.add(
       LSTM(300, dropout=0.2, return_sequences=False, input_shape=(None, 300)))
   self.model.add(Dense(2, input_dim=300))
   self.model.add(Activation("linear"))
   self.model.compile(loss="mean_squared_error", optimizer="rmsprop")
   

But the model seems completely incapable of learning to identify the QRS complex.

Does anyone have any advice of how I might improve things. I'm expecting to get more data, but actually I'm not sure that will help here as I'm certainly not over-fitting the data I've already got.

Do you think resampling my data series at regular intervals so it's just voltage in a linear time series might help, so my input is 300 x 200 x 1, rather than 300 x 200 x 1? It's not ideal as the sampling frequency changes, but I am willing to try if people think it will help.

EDIT

Just in case anyone is interested, I made a few changes to my network and now it works beautifully:

Red curve = Probability of being QRS onset

Green curve = Probability of being QRS offset

Dotted lines are maximum probability for each

Basically the changes I made were

1. I got rid of the time series array, so the input data was just an array of voltage data
2. I instead made two networks, one for QRS onset and one for QRS offset; this appears to have reduced the tendency for the network to find one of the parameters, and then just place the other based of the average QRS duration.
3. I changed the task for a regression task to a classification task, so each data point had a probability of being QRS onset (thanks @shimao)

Answer 1

I would suggest framing this as a classification problem and outputting 2 softmaxes each with size 300. This usually works better than the continuous output approach you have taken here.

You might expect this approach to work better, because in order for the LSTM to successfully execute the original regression approach, it would have to detect the onset, and then somehow pass down that information several hundred time-steps. In addition, there would probably have to be a counter-like mechanism embedded in the LSTM weights in order to figure out exactly where the deteted onset was. This is all super difficult for an LSTM to learn to do.

Also for that reason, I don't recommend just taking the hidden vector from the last time-step of LSTM and getting the output from that -- instead, try doing something with the full sequence of hidden states (flatten them or something).

Answer 2

I tried to do this also and got stuck at exactly the same point. I made my solution available on github (https://github.com/niekverw/Deep-Learning-Based-ECG-Annotator). After reading seq2seq work of semantic segmentation of pictures, I realized that you could apply it to ECG with LSTMs as well to segment the ECG with categorical_crossentropy. This has the advantage (I think) that there is much more for the model to learn, and you don't need >1 model.

model = Sequential()
model.add(Dense(32,W_regularizer=regularizers.l2(l=0.01), input_shape=(seqlength, features)))
model.add(Bidirectional(LSTM(32, return_sequences=True)))#, input_shape=(seqlength, features)) ) ### bidirectional ---><---
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(Dense(64, activation='relu',W_regularizer=regularizers.l2(l=0.01)))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(Dense(dimout, activation='softmax'))
adam = optimizers.adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy']) 

• Colors at the top show true annotations
• Colors at the bottom show predicted annotations