Leave one out cross validation for LSTM

I have a multivariate time series data set having 6 variables. I have to predict the sixth variable at the next time step given the expected values of other five at the next time step. I am using LSTM for prediction. The data set is small (170*6), have 170 observations. Splitting the data set into three i.e. train/test/split doesn't seem a probable option in this case.

As far as I know, input sequences (of fixed length) over which the LSTM learns are assumed independent. So my question was, can one sequence be left out for validation just like in the case of a cross sectional data problem.

Leave one out cross validation for LSTM, or any other time-series model, doesn't really make much sense because it would introduce missing values in the series and leaking information from the future.

Time series models learn from historical values, to predict future. In leave one out cross-validation, you remove observations from the series, including the past ones, what leads to missing data. What follows, sch cross-validation strategy would need the model to be able to deal somehow with missing data, or would need some external strategy of dealing with them and basically would test how well the model deals with missing data. Simple cross-validation can be used only if the samples can be assumed to be independent, this is never the case for time series (unless you assume it to be random noise).

With time series data, as described by Hyndman and Athanasopoulos (2018), we perform one step ahead cross validation, where we use $$1,\dots,k$$ samples, to predict $$k+1$$ value (or alternatively $$k+1,\dots, k+m$$ values), where we use different $$k$$ values, as illustrated on image below (also by Hyndman and Athanasopoulos, 2018).

Such strategy tests what you are trying to test, i.e. how well does your model predict future from the past. Moreover, it also tests how well does the model deal with changing data, a scenario that is very common in such cases, since you usually don't make single prediction using static training sample, but want a model that would be re-trained with new data and would adapt.

Commenting on LSTM networks

Input sequences (of fixed length) over which the LSTM learns are assumed independent. So my question was, can one sequence be left out for validation just like in the case of a cross sectional data problem.

let me comment on this. The above answer applies to LSTM network as well. LSTM network learns the time-dependence in your data by using the sliding LSTM window, so the sequences are not assumed as independent, as the idea of "long term memory" is the dependence assumption. Usually the sequences are generated as in this Keras code example (see also this tutorial for code with description):

sentences = []
next_chars = []
for i in range(0, len(text) - maxlen, step):
sentences.append(text[i: i + maxlen])
next_chars.append(text[i + maxlen])


As you can see, the loop is used to iterate through your data and create a sliding window, that makes jumps of step size, where step is smaller then maxlen. So the sequences are dependent. Even if you used step equal to maxlen, still you would be assuming time-dependence by choice of the model.

What follows, dropping one of the sequences leads to introducing missing data. Using the simple cross-validation would introduce bias, since you would be measuring how does your model work in missing data scenario, rather then how well does it make predictions for the future. For time-series, you should rather use the one step ahead cross-validation.

• thanks! I appreciate the technique you mentioned in the answer and I have edited the question to reflect my understanding of LSTM for time series. Please correct me if I am wrong. – naive Jun 17 '18 at 8:14