# Why is my LSTM +- 1DConvNet so ineffective at waveform analysis?

I'm trying to learn LSTMs and I thought a nice way of doing it would be identifying onset-and-offset of QRS complexes on ECGs.

I have 300 x 200 x 2 numpy array of ECGs (300 ECGs, each of 200 data points, each data point being [x=time,y=voltage]. Sometimes it's 0 padded at the start (like the example below) as ECGs can be of different durations.

I also have labels in the form 300 x 2, 300 labels of [qrs_onset,qrs_offset].

This is an example of an ECG plotted and labels superimposed.

I've done some pre-processing, shuffling the location of the ECG forwards and backwards by ~ 10% of the cycle length. I then feed it into an LSTM model with Python/Keras/Tensorflow backend:

self.model = Sequential()
self.model.compile(loss="mean_squared_error", optimizer="rmsprop")


However, all the LSTM does is fine a location that fits the entire dataset best, and gives that exact location regardless of the ECG fed to it.

It takes about 4000 epochs to get to this point, but at this stage the loss completely plateaus and makes 0 further progress.

It's strange because I thought an LSTM would be great for this task.

Things I have tried:

1. Reducing the epoch number to a smaller proportion of the dataset to get out of local minima
2. Adding between 1 and 3 LSTM layers of between 64 and 300 units.
4. Adding and removing dropout between 0.0 and 0.2
5. Adding a 1D Conv layer to try and identify the peaks using spatial information before feeding this into an LSTM

self.model = Sequential()
LSTM(300, dropout=0.2, return_sequences=True, input_shape=(None, 300)))
LSTM(300, dropout=0.2, return_sequences=True, input_shape=(None, 300)))
LSTM(300, dropout=0.2, return_sequences=False, input_shape=(None, 300)))
self.model.compile(loss="mean_squared_error", optimizer="rmsprop")


But the model seems completely incapable of learning to identify the QRS complex.

Does anyone have any advice of how I might improve things. I'm expecting to get more data, but actually I'm not sure that will help here as I'm certainly not over-fitting the data I've already got.

Do you think resampling my data series at regular intervals so it's just voltage in a linear time series might help, so my input is 300 x 200 x 1, rather than 300 x 200 x 1? It's not ideal as the sampling frequency changes, but I am willing to try if people think it will help.

EDIT

Just in case anyone is interested, I made a few changes to my network and now it works beautifully:

Red curve = Probability of being QRS onset

Green curve = Probability of being QRS offset

Dotted lines are maximum probability for each

Basically the changes I made were

1. I got rid of the time series array, so the input data was just an array of voltage data
2. I instead made two networks, one for QRS onset and one for QRS offset; this appears to have reduced the tendency for the network to find one of the parameters, and then just place the other based of the average QRS duration.
3. I changed the task for a regression task to a classification task, so each data point had a probability of being QRS onset (thanks @shimao)
• I've had similar problems and in the end it usually ends up being a problem with input / output while training. – Joonatan Samuel Sep 27 '17 at 10:55
• Would be helpful to show the changes you made in order to make it work. – tafelplankje Dec 30 '17 at 15:13

I would suggest framing this as a classification problem and outputting 2 softmaxes each with size 300. This usually works better than the continuous output approach you have taken here.

You might expect this approach to work better, because in order for the LSTM to successfully execute the original regression approach, it would have to detect the onset, and then somehow pass down that information several hundred time-steps. In addition, there would probably have to be a counter-like mechanism embedded in the LSTM weights in order to figure out exactly where the deteted onset was. This is all super difficult for an LSTM to learn to do.

Also for that reason, I don't recommend just taking the hidden vector from the last time-step of LSTM and getting the output from that -- instead, try doing something with the full sequence of hidden states (flatten them or something).

• That's a great idea, thank you! Can you explain or give an example of what you mean flattening the hidden states rather than doing what I've done? I don't quite follow how you'd use that as an output and calculate loss etc. ... – James Sep 28 '17 at 7:48
• So if the hidden state has dimension 64 and there are 300 time-steps, instead of taking the last hidden vector, which has size 64, you might want to take all the hidden vectors by setting return_sequences = True. I imagine this will give you something shaped 300x64, which you can then reshape into a 19200 dimensional vector, and then add some linear layers on top of that. Since 19200 is rather large, I suggest that the last LSTM layer have a smaller output dimension (perhaps 8 or so). – shimao Sep 28 '17 at 12:39
• Thank you, I implemented it as demonstrated in my post and it works beautifully – James Sep 29 '17 at 7:53

I tried to do this also and got stuck at exactly the same point. I made my solution available on github (https://github.com/niekverw/Deep-Learning-Based-ECG-Annotator). After reading seq2seq work of semantic segmentation of pictures, I realized that you could apply it to ECG with LSTMs as well to segment the ECG with categorical_crossentropy. This has the advantage (I think) that there is much more for the model to learn, and you don't need >1 model.

model = Sequential()