Using 10 fold CV for performance estimation of a logistic regression model, what is the appropriate way to incorporate multiple imputation for missingness across the predictors and outcome in which the mechanism is assumed to be missing at random? Also, do we include the outcome in the imputation models such that predictor variables can be used to impute the outcome and vice versa, or should we not consider the outcome in the imputation models?

This is what I am thinking: Using the training data only (90%), perform multiple imputation 10 times, then fit a logistic regression model to each imputed dataset, then average model coefficients across the 10 imputed datasets to obtain a single logistic regression model with averaged coefficients. Using the test data only (10%), perform multiple imputation 10 times, then fit the 'average' logistic regression model obtained from the multiply imputed training data to each of the 10 imputed test datasets, then average the error across the 10 imputed test datasets to obtain the average error corresponding to the averaged model. Repeat this process 10 times (i.e., 10 fold CV) such that in each fold there is a single average model derived from the multiply imputed training data and a single average error of this model derived from applying the model to the multiply imputed test data. Then average the 10 average errors (1 from each fold) to obtain the final performance estimate.


1 Answer 1


I believe that your thinking is right.

The alternative is to perform multiple imputation on the entire dataset prior to splitting into train/test partitions. Doing so would mean that some information from the training sets is used to create/impute values in the test sets. In other words, there would be leakage from the training sets into the test sets, thereby biasing the results of cross-validation.

  • $\begingroup$ @pauluccd927 does this answer your question ? If so, please could you mark it as the accepted answer. $\endgroup$ Commented Dec 2, 2019 at 16:32
  • $\begingroup$ I do agree that the alternative would result in leaking from training and tests and bias the CV. However, I am not sure if I can at this time accept the answer that you think my intuition is right, being that my question is whether my thinking is right. $\endgroup$ Commented Jun 29, 2020 at 17:01
  • $\begingroup$ @pauluccd927 I have updated my answer :) $\endgroup$ Commented Jun 29, 2020 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.