Assume I use a moving window to slice a daily stock closing price history data. Using past 7 days to predict next day. For each training instance, I'm strictly using historical data to predict future event. However, after processing the data to many many small trunks each with 7 day -> 1day (feature-> label) pair, if I discard the actual date and mix all the records, and then split the data into training and testing, it is typically considered data leakage, right? I am wondering why. Assuming the data is trained on a DNN. For each forward and backward pass, the loss is only consider the feature+label which doesn't have leakage. When and where is the leakage happening?
4
-
1$\begingroup$ You are describing two different training strategies: (1) moving windows, (2) mixing all records, followed by splitting. What do you mean by "after processing the data"? How would you first train your model using strategy 1, then strategy 2? Please clarify. $\endgroup$– Stephan KolassaCommented Nov 8 at 8:51
-
1$\begingroup$ That said, temporal leakage occurs if there are temporal dynamics, such as autocorrelation. Today's weather is correlated with yesterday's and tomorrow's. A simple random split will learn today's weather from yesterday's and tomorrow's. This is leakage, because of course in an actual production case, we won't have tomorrow's weather as a feature to predict today's. $\endgroup$– Stephan KolassaCommented Nov 8 at 8:51
-
$\begingroup$ there is no strategy 1. moving window is just for feature processing. using a moving window of 8 days to scan the time series and prepare input features and target. my question is about how the leakage happens? The whole idea of ML is to assume historical patterns can predict future events. For each instance, there is no leakage. Why aggregating with different times introduce leakage? The loss didn't consider any future information. $\endgroup$– yangCommented Nov 8 at 15:58
-
$\begingroup$ Ah. Thank you for your clarification. Then I believe my second comment above answers your question. I will post it as such. $\endgroup$– Stephan KolassaCommented Nov 8 at 16:12
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
Temporal leakage occurs if data is randomly split for training if there are temporal dynamics, such as autocorrelation.
As an example, today's weather is correlated with yesterday's and tomorrow's. A simple random split will learn today's weather from yesterday's and tomorrow's. This is leakage, because of course in an actual production case, we won't have tomorrow's weather as a feature to predict today's.
As a different example, consider credit scoring. If we use applicants' entire history, we would "learn" that someone is more/less likely to default based on information after the event (the decision to extend credit, and on what conditions). Someone who loses their job two years after getting credit is more likely to default half a year later. But including job loss two years out in a model that is trained to predict credit default 30 months out simulates having a crystal ball: this is leakage.
In any situation where time dynamics can be suspected, one should make sure to always only use features previous to the event to be trained or decision to be supported.
answered Nov 8 at 16:17
-
$\begingroup$ keep in mind that I assume training data is strictly prepared with historical price as input and future price as target. In your credit example, it is equivalent to for each training instance, you can only use user's history events to predict future events. For each instance, there is no data leakage. My question is about why when aggregate these no leakage instances, there is data leakage? $\endgroup$– yangCommented Nov 8 at 17:11
-
$\begingroup$ I think I stll do not understand what exactly you mean by "aggregate". Whether your features are engineered using only "past" information is irrelevant. If during training, you use features that contain "future" information, or use "future" actuals (which your model may use as "future" features), you have leakage. $\endgroup$ Commented Nov 8 at 17:45
-
$\begingroup$ the point is "feature" comparing to target never use future information. For example: input(d1-d7):target(d8), input(d8-d14):target(d15), input(d3-d9):target(d10), etc. There is no leakage if you check any single data point. However, if I assign input(d1-d7):target(d8), input(d8-d14):target(d15) in training set, and input(d3-d9):target(d10) in testing set, people start to say there is data leakage since I have "future data" in training compare to testing. Why? $\endgroup$– yangCommented Nov 8 at 18:02