7
$\begingroup$

I was reading this post about a reproducibility tool when I realized that the author decided to split the train/test datasets before featurization.

It didn't make sense to me, because why would you not be able to commute those operations? But just a few days later I found this comment in a deepchem repo issue:

In cases of data augmentation, there are use-cases when it might make sense to featurize data after it has been split into train/test/validation.

And I have no idea how that would be a thing, but apparently it is. What kind of data augmentation - or any other data transformation for that matter - could justify postponing the dataset splitting?

$\endgroup$
2
  • $\begingroup$ You are correct to question this (+1). No real data transformation/feature extraction is justified to occur prior to dataset splitting. Data leakage is a serious problem in many ML pipelines exactly because this point is not fully appreciated. $\endgroup$
    – usεr11852
    Apr 3, 2018 at 21:31
  • $\begingroup$ It really depends on what preprocessing you are doing. If you try to estimate some parameters from your data, such as mean and std, for sure you have to split first. If you want to do non estimating transforms such as logs you can also split after $\endgroup$
    – 3nomis
    Dec 29, 2019 at 15:39

1 Answer 1

9
$\begingroup$

That comment is correct: we have to do "feature extraction" from our training data only.

Let's consider one of the most common data-transformation procedures, centring. We get an "expected value" $\hat{\mu}_{x_j}$ for our feature $x_j$ and then we subtract that from the values of $x_j$, nothing magical. A central question is: what this "expected value" reflects; does it reflect our understanding of $x_j$ using the whole sample or just the training sample? If we use the whole sample we have what is called data-leakage, "we cheat" using information that should be available during prediction. To give an NLP example, if some very unusual word or n-gram is present in our corpus and all instances happen to land in the test set, it is obviously wrong to inform the process of convert our collection of text documents to a matrix of token counts with that unusual n-gram. It will give us a false scene of security about the generalised performance of our procedure. Therefore, when doing serious feature extraction/engineering we need to use only the training data and not the whole dataset.

$\endgroup$
2
  • $\begingroup$ Wow, I'm certainly guilty of leaking future info into train data because of this. Thanks! :) $\endgroup$
    – villasv
    Apr 3, 2018 at 21:35
  • $\begingroup$ No problem. I am glad I could help. It is a slightly subtle matter at first but when you work around it a bit becomes almost rudimentary. I made similar mistakes when I started off. $\endgroup$
    – usεr11852
    Apr 3, 2018 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.