For the dataset, I know that:

  • for missing values in training dataset (and therefore for validation datasets for CV) we impute values using training samples
  • for missing values in test dataset we impute values using test dataset only to avoid polluting the test data with our traning data

What should I do with new values, that I get in the actual deployment? For example, in medical application doctor measured only 10 out of 34 values for Breast Cancer Wisconsin - which data should I use for imputation?

As I see it, our whole dataset (nevermind the division into training and test) is just a statistical random sample of N IID individual samples from some problem space. Therefore I think I should use whole dataset for data imputation, since I expect other data to have roughly the same statistical properties, and my whole dataset is the best estimate (largest random space sample) that I have available. Am I right?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.