I have been thinking about the way of splitting the data into training, validation and test sets for the stateless LSTM. For me, the intuitive way is to arrange the original data into the 3D form (batch_size, time_step, variable_number), then randomly split it into 90%, 5%, and 5%. However, both validation set and test set have a lot of data points overlapping to the ones in the training set even though they belong to different sequences. I guess it is OK for the validation set, but not for the test set because many data points have been used to build the model. If so, should I randomly separate 5% consecutive data points from the original data as the test set before arranging the original data to the 3D form? Note: @Tim♦ The question you demonstrated is more about stateful LSTM and cross-validation. I am asking the questions about the validation for the stateless LSTM.

  • $\begingroup$ The linked question seems to answer your question. LSTM models dependence in time, so obviously you cannot use validation set randomly because it would leak information from the future. It is bad for both training and test set. For validating time-series models we usually split the data on time dimension, so e.g. we take 80% of the "past" as training set and 20% "future" as test set. If the linked thread does not answer your question, please edit it to make it clear what makes it distinct. $\endgroup$ – Tim Dec 17 '18 at 13:09
  • $\begingroup$ Please edit your question to explain what is your model and your data. Otherwise it is unclear what exactly is your setting and suggest appropriate cross-validation split. $\endgroup$ – Tim Dec 17 '18 at 13:29
  • $\begingroup$ @Tim♦ As I know, for the stateless LSTM the sequences are independent from each other. This means they can be shuffled and selected randomly. Keras has an argument 'stateful' for LSTM layer. Only when stateful=True the sequences are correlated in some sequential way. $\endgroup$ – Dong Dec 17 '18 at 15:20
  • $\begingroup$ So if your sequences are independent of each other (are they?), what exactly is the problem? If they are unrelated whatsoever, you can just sample them randomly. $\endgroup$ – Tim Dec 17 '18 at 15:23
  • $\begingroup$ @Tim♦ Actually they are not independent of each other. Just because I want to focus on the sequential relation within every sequence rather than the relation among the sequences, I leave stateful=False to assume there is no relation among sequences. In this way, there are a lot of data points appearing in all the data sets (training, validation, testing). This makes the validation and testing sets not good (especially for the testing set), I guess. On the other hand, the same points belong to different sequences in different sets. (to be continued) $\endgroup$ – Dong Dec 17 '18 at 17:22

Browse other questions tagged or ask your own question.