2
$\begingroup$

I'm trying to perform binary classification on a dataset with missing values. I used sklearn's iterative imputer to impute these values and I got pretty good results. However, I realized that I was performing the imputation on a dataframe that contained the output variable. This seems wrong to me, but when I googled it I found several sources that said the outcome variable should be included. I would like to believe this because my model performs much better when I do the imputation with the outcome variable, but it doesn't make sense to me. If I include the outcome variable how am I supposed to apply this imputation to new data which doesn't have the outcome variable yet?

$\endgroup$

1 Answer 1

4
$\begingroup$

It's a simple result of Bayes rule. If the outcome variable is conditionally dependent on any of the inputs, then the imputation model for the inputs will not be correct unless it includes the outcome as a feature.

To impute missing data in a new data frame, you have to train an imputation model which includes missing outcome as a missingness pattern. In that case, supposing you are using MICE as it's one of the most widely implemented multiple imputation platforms out there, you can impute the whole dataset, including the outcome, and use that information to perform prediction, simulation, or any number of tasks. However, once the actual outcome is observed, you should replace the "missing" column of outcomes and redo the imputations (because you now observed the outcome), that way you get efficient inference.

Or you can omit the outcome from imputation. This marginal approach will increase the MSE of predictions by incorporating some bias, and less efficient inference, but that in and of itself is not a reason to exclude a method if the actual scientific question implies it.

$\endgroup$
1
  • 1
    $\begingroup$ Just to clarify, if I have a train and test dataset and I want to do multiple imputation I should impute the whole train dataset, including the outcome, and then use this imputation on the test dataset with its outcome variables marked as missing? Then replace the imputed outcome variable of the test set with the actual outcome variable to test my model? Thanks $\endgroup$
    – Clay James
    May 22, 2019 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.