Data leakage in temporally overlapping train-test split

Question

Question: In the sliding window train-test split strategy, will there be data leakage if, say, I train on a dataset $X_{t}$ to predict values $y_t$ that were collected after my test data $X_{t+1}$?

Background: I'm trying to predict whether returns on investments for companies in a portfolio on a twelve month horizon will do better or worse than average (binary classification).

I have 100,000 historical monthly observations of companies financial information (a hundred features like market cap, daily liquidity,...) taken at one month intervals, companies in each interval are not necessarily the same.

I have tried splitting the data into train and test but they had different distributions and models were performing badly (~51%). So I did a sliding split, training on a given month $t$ and testing on the following one and had good results doing so (80%).

But I was told by a more experienced statistician, that I should use test data collected after returns on month $t$ were known. Meaning the test set should be the data collected in month $t+13$ or later.

But I don't see where the problem is since I don't have access to returns in my training data.

Answer 1

I agree with you and I see no information contamination from the test into the training. You do not use any of the xtest sets into the training. The most one can claim is that probably the Xtest is not that different that Xtrain, and the same for the ytest and ytrain given the autocorrelations of naturally occurring time series. If you can use the autocorrelation when the system is into operation, there is no problem. That mean that you have to use the xtest/ytest that you just predicted as the new xtrain/ytrain for the next prediction. You have to retrain after each prediction.

If, other other hand, you cannot retrain the system, then the advise given to you is semi-correct, but the t+13 seems arbitrary, unless the experienced statistician knows that after 13 intervals the autocorrelation of the time series is very low.

TLDR: if you can retrain after each/a few predictions your schema is OK, but know that you are probably benefiting from the autocorrelation of time series and if you use less correlated segments for test, your predictions should be of less quality.

Answer 2

In a real production environment, if you want to predict ytest using Xtest, you won't have access to ytrain as you did in your experiment. The statistician's suggestion mimics reality better and is expected to provide more reliable predictive performance. With that said, is your approach not reliable? Not necessarily. I think the reliability depends on the interplay among ytrain, Xtest, and ytest. To defend your approach, I feel it's easier to run the experiment your statistician friend suggested and see if there is still a good result, rather than to argue along the line of no data contamination from ytrain into Xtest and ytest. Just my two cents.