0
$\begingroup$

When dealing with missing values, I see some people calculate some value on train set (mean, median, zero, etc) and use that value to fill the missing values on both the train set and the test set.

Also, some people combine the train set and the test set and calculate some value on the combined set to fill the missing values.

Is there a correct way? Or both are ok?

$\endgroup$
1
$\begingroup$

There is a correct way to do it.

First, you should never touch/work with your test set while training your model. The test set shall be used to assess your final model's performance. Thus, you should "forget" about it and treat is as unseen data.

Hence, when you fill your missing values in the test set, you can use the mean, median, etc. from the train set. If your train set is reasonably constructed and not biased (e.g. toward single class) - then this data imputation method for the test set will be fairly ok.

To answer your question: No, not both are ok but only one way is ok.

I think if you read more about data imputation, it will help you to understand more about the topic.

There is a lot of research going on in data imputation for missing values. Here, it is valuable to know, that researchers differentiate between different types of missing values:

  1. Missing completely at random (MCAR)
  2. Missing at random (MAR)
  3. Missing not at random (MNAR)

It is important to differentiate between these cases by understanding the source of missingness before deciding what method to use.

Some pointers for you:

https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779

http://www.lcc.uma.es/~lfranco/A27-Jerez%2BMolina%2B%2B2010.pdf

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the explanation. In some Kaggle kernels, people combine the training set and test set. Then, they fill the missing value using all of them. The argument is that the test set and train set are originally one set, so they could do this... I disagree with this method. Do you think this could be justified? $\endgroup$ – JoeB Jul 28 '19 at 18:41
  • $\begingroup$ @SolingerMUC: the OP is about imputation of an independent test set. Could you elaborate on how to do this in a non-MCAR situation, e.g. if all variables contain MAR missings? $\endgroup$ – Michael M Jul 28 '19 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.