what does it mean that there is leakage of information when one uses a test set?

Question

I have read about the term "leakage of information" that occurs when one tries to estimate the generalization error by using a test set in Machine Learning models. However, I was not able to find any formalism (maths or statistics) that could help me to clarify why this situation happens. Could anybody explain to what does this term refers to?

Answer 1

Data leakage occurs when there is information in your test set's predictors that wouldn't be available when the model is "live." There are egregious and subtle cases of data leakage.

Egregious case. Say the goal is predicting retention of an insurance policy during the first year. At month 3 there is a scheduled check-in with a company representative, and after the check-in the data element had_check_in flips from False to True. A junior modeler is working on a cross-sectional data set (no time dimension) with information from the last two years, and has_check_in is one of the variables. The modeler concludes that this variable is very important, because when it is True, the policy holder is more likely to keep the policy throughout the period of study. Clearly that contains information from the future, and in a live run of the model, all had_check_in values would be False for new cohorts!

Subtle case. Suppose that now the junior modeler is approaching the above problem with a time dimension, having learned from the last mistake. He takes a holdout set of 2000 policy holders (across all time) and use the remaining policy holders' retention values to build a model that, among other variables, uses month and year. Then he runs the predictions on this test set to get holdout metrics. While this is unlikely to be a disaster, there's information leakage in that aspects of the particular months and years can be learned from the members of the training set. In a prediction scenario, you couldn't estimate the properties of the future time period from actual policy holders, so the holdout metrics are likely to be optimistic.

I've seen both cases in practice, but the subtle case is much more common. It makes me hesitant to use automatic cross validation routines from sklearn, etc., because I feel these situations need to be carefully thought out in general.