I am trying to build a recurrent neural net model using lstm trying to predict future outputs from a financial time series. Outputs are classified in macro classes according to the magnitude of absolute changes. My inputs are the variable i am trying to predict itself plus some other features. I build windows of inputs each one representing a sample, The window size is fixed and equal to the chosen timestep of the model. So each samples is an ordered sequence of timestep features. The approach i am thinking to use for data preparation Is between the following two: Approach A 1) split the initial dataset so that first 80% of observations is my training set and the last 20% Is my final test set 2) randomly shuffle the first set and then divide It in two parts, training and validation for hyper parameters selection. Also I would choose the training set so that output classes are equally represented in the training (i.e. Say i have three output classes, i drop from the training set the samples associated with the two most frequent labels in excess of the least represented so that labels are 33% each). Note: The shuffle involves the batches of sequences of course, so each sequence would still preserve its own internal order. 3) standardize the three sets (shuffled training set, shuffled validation set, unshuffled ordered test set) according to the first training set magnitude
Approach B 1) shuffle the whole dataset as first thing (of course I mean shuffle the batches of sequences, each one would still be ordered in its inside) 2) splitting It in three parts, training validation and test sets using same stratification approach described above 3) standardize as in approach A
Which of the two would you think could be a more robust approach? Please bear in mind I am not using a stateful kind of model. My doubt Is that, because of the sequence nature of each sample, by shuffling samples the test set will contain some "deja-vu" data that have already seen during learning thus leading to misleading expectations when tested on future realizations.
Thanks for your opinions