0
$\begingroup$

I've consulted other posts here regarding imputation and the recommendation seems to be to not impute on unpartitioned data. Data should be partitioned and missing values in both train and test set should be imputed from the mean (or whatever imputation method is chosen) computed out of training set.

What I'm trying to understand is why not impute missing values in train set from train set itself, and those in test set from test set?

By imputing test set from the training set, you're allowing test data to be influenced by training data and thus this should increase the risk of overfitting.

$\endgroup$
1
$\begingroup$

Look at this the other way. The unlabelled data, you'll be working with in future, may come with missing features. To handle it, you'll be using your transformer. While training, you want to mimic this behaviour. That's exactly what will happend on prod. Why should you want to impute values of test set from itself? This experiment won't tell you anything about model's behaviour on prod.

$\endgroup$
  • $\begingroup$ That's a fair point. $\endgroup$ – Kshitiz Sharma Nov 18 '19 at 16:07
  • $\begingroup$ Furthermore, it's not only about imputation. You should treat discretization, rescaling, resampling, dimensionality reduction the same way. You train pipeline of transformers on your train set, and later apply them to your test data. Whole point of validation is to gain some insight to your model, before you send it to prod. If you're curious you can always check implementation of Pipeline from sklearn. It's on github. $\endgroup$ – Piotr Rarus - Reinstate Monica Nov 18 '19 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.