Imagine the following problem. I have weekly snapshots of price data of K items, as well as of various features/predictors. I want to predict how much the price will change 2 years from now.
I assemble my dataset as follows: each row consists of features for each item for each week, and the output variable is forward 2 year price returns. The date of the observation is not in my dataset - I only use it to separate the dataset into a training and validation set, i.e. in cross-validation (where I discard 2 years of data before and after the validation time-period (which is 1 year) to prevent data snooping).
Clearly, the samples from two consecutive weeks for the same item (and even for different items) will be highly correlated, both in terms of the features and in terms of the response variable (as the forward 2 years will largely overlap, meaning the returns will be very similar). What potential problems can it cause for supervised learning approaches, e.g. random forests or gradient boosted trees?
My thoughts are:
- The effective size of the dataset will be smaller than expected. I.e. my dataset of, say, 100'000 observations will behave more like a dataset of 100'000 / (52*2) ~= 1000 observations, because that is the number of samples for which response will not have autocorrelation. That will significantly affect the complexity of models that I can fit to the data, i.e. I will have significant overfitting problems and have much poorer results than expected.
- Because of features being very near each other on consecutive weeks for each item in the feature space, my dataset will cover the feature space a lot worse than expected, again reducing the "effective" size of the dataset.
- Using only 1 year of data for validation in cross-validation will result in high variance of cross-validation results, because once again, the effective number of samples in the validation set will be ~K rather than 52*K.
Are these valid concerns? If yes, does it mean that with K~=100, I will need hundreds, if not thousands years of data to be able to train a reasonably complex non-linear model from hundreds of features, e.g. using random forests or gradient boosted trees? Or am I being over-pessimistic and my argument about "effective dataset size" above is nonsensical?