Training a RNN on time series: How to cope with different sequence origins?

Question

I am wondering if I should apply a recurrent neural network on my data. Data is EEG from sleep, and thus there is much information hidden in the temporal domain. Ergo, RNNs make sense.

Intro: I have calculated and selected a set of features from the time signal, and averaged over samples so that each input to the network will be a vector and represent a second of sleep.

Concern No. 1. Data is from several hundred different subjects, and naturally each subject will have a different nature of EEG. I will normalize the data of each individual subject to have zero-mean unit-variance.

In a regular feed-forward neural network, I would randomize the order of inputs to speed up the training, but I guess that is not feasible with RNN due to the memory?

Concern No. 2. Due to artefacts/noise in data, I might want to throw away certain segments in time of the data. Therefore, my input will not always be sequential. It can be illustrated like

1 2 3 4 5 6 7
+ + - - + + +   (+ are included in training data, - are thrown away)

Thus, the data at time index 5 will have the memory of time index 2 and not 4.

How do I cope with this? Will it just be considered noise? I imagine I am not the first in the world to consider this issue.

Answer 1

You didn't say which type of RNN you use: LTSM (context layer and gates) or classic(only context layer) RNN.

Concern 1: I use in my work, batches of 128 sequences and used sequence sizes of 100 time points. When I test the dataset I use batches of 1000 sequences of size 100 time points, and I don't have problems with memory. You size your batch as much memory you have, and this applies to any kind of neural net..recurrent, convolutional, etc.

Concern 2: This is problem of missing/corrupt data in a dataset. I wouldn't remove the time points from the sequence just because the data is corrupt, but fix the data. One easy way to fix the data is to compute an average for that feature on the entire dataset, and put it in the dataset where it is missing or corrupt.

Answer 2

1) By randomizing the order of inputs I assume you mean shuffling the sample vectors from different people and classes - which is something you should definitely do, so as to avoid skewing training towards a certain person or class from sequential samples.

Why would shuffling the training data be memory hungry? You could shuffle before starting training. If you want different shuffles per epoch, you can do it too, without considerable overhead (eg, if you're using tensorflow, it has a built in threaded queue for shuffling and feeding the training data. See here)

2)If you want the model to generalize, you should leave the noise where it is, if it is not catastrophic. If it is, you could fix the dataset as Chelaru says, or remove the destroyed portions and split up the relevant vectors. I.e., from your example:

1 2 3 4 5 6 7 (sample #1)
+ + - - + + +   (+ are included in training data, - are thrown away)
split:
1 2 (sample #1.1)
5 6 7 (sample #1.2)

This introduces other issues, such as minimum meaningful vector length, potentially necessary padding, etc.

Answer 3

1

If by order of inputs you mean the ordering by the time of observations of subjects, then, the answer is, of course, NO, you cannot randomize the time of observations. Otherwise, your data would not be called time series, then you wouldn't need to bother about time at all. It's highly unlikely to be the case for EEG, in my opinion. Usually, you can randomize the order of subjects.

2

Skipping observations can be a problem. In time series analysis there are models such as state space models (SSM) where skipping/missing observations of this type is not a big issue because of the way it is handled.

For instance, in SSM you model the unobserved state and observed measurement. When you skip the observed measurement, the unobserved state still keeps updating. You can estimate the state, then infer would would have been the measurement too.

On the other hand, sometimes skipping observations is a common practice. For instance, in finance we often collect trading indicators only during the business days skipping weekends and holiday. It is a common practice to treat the changes from Mon to Tue the same way as changes from Fri to Mon. It depends on the problem at hand.

So, you have to think about whether this would be appropriate for your case. If it is not then maybe you need to tweak the formulation of your model so that it explicitly takes care of the skipping data. One way to do this would be calculating the difference between Mon and Fri then scaling it to one day difference by dividing by 3. This way each observation will represent one day change, although Mon observation will have more noise in it than Thu or Wed.