Data preparation (preprocessing and data cleaning) before or after train-test split with scikit learn?

Question

I have been practicing Data Science/Machine Learning, and I am confused about when to complete the following tasks when using train-test split in scikit learn:

1. EDA
2. Filling in missing data
3. Removing data that isn't needed
4. Working with categorical data
5. Feature Engineering
6. Feature Selection

I know that scaling should be done after train-test split. However, I have seen both approaches (train-test split before and after) when working with the above tasks.

I'm confused about which is the correct way for these tasks.

Can someone provide some clarification?

Thank you!

Answer 1

1. Test set is supposed to be untouched until the final stage: test data insights should not affect our decisions since it simulates the data from the production stage. However, if we split it ourselves, we assume the distributions having no significant differences, so exploring the full data is not technically a mistake.
2. Filling with a constant should create no leakage. Strategies like filling with mean values result in a leakage (albeit a minor one, if it's a school project, reviewers are known to often turn a blind eye on that).
3. Depends on the usability criterion. If it's truly, absolutely never going to serve as a model input, there is no difference. Dropping outliers before splitting is somewhat cheating, since that excludes the hard to predict samples from the test, - we should only do this if we expect the real input data to be filtered the same way.
4. Depends on the encoding strategy. OHE shouldn't result in a leakage, target encoding would make a huge leak if done before splitting.
5. If we operate within a single observation (like, adding two features' ratio), there's no leak. If it's based on other observations somehow, it's a minor leak. If it considers target too (e.g. SymbolicTransformer()), it's a major leak.
6. Once again, if we assume the distributions to be roughly the same, stats like mutual information or variance inflation factor should also remain roughly the same. Generally, I'd stick to selection using the train set just to be sure.

TL;DR: when in doubt, don't touch the test set until your best model is ready to be tested.