Imputation when estimating prediction error

Question

I'm working on estimating the performance of a predictive model. However, a rather large amount of covariate data is missing. My initial idea was to use cross-validation to estimate the prediction error by doing the following in each step:

1. Fit an imputation model on the training part
2. Use the imputed training data to fit a predictive model
3. Impute the test data using the imputation model from the first step
4. Use the imputed test data to get an estimate of the prediction error.

When building the imputation model I would include the outcome so that the imputed values in the training set are not independent of the outcome. However, my gut-feeling tells me that using the outcome to impute the test data might lead to an optimistic estimate of the prediction error. I was wondering if this would be the standard approach to take or if there's some literature available that deals with this problem?

Answer 1

First, unless you have many thousands of cases, breaking your data into separate training and test data sets is likely to lose considerable power. You are better off using all the data together and then bootstrapping all the steps of your model-building process (including imputation) to estimate the reliability of the process.

Second, it is generally better to use multiple parallel imputations rather than repeated serial single imputations (as you seem to propose) to deal with missing data. When done properly, on data where values are missing at random in the technical sense (not necessarily "completely at random"), such imputations can include outcome variables in the imputation model. This thread on how to incorporate multiple imputation into a model-building process includes an answer from Stef van Buuren, an expert on multiple imputation.

Third, you should consider why you wish to model based on data with so many missing values. If values of some particular predictors are hard to obtain and will be hard to use in the future, what do you gain by including them in you model?