Der community
I have a question about the appropriate handling of the imputation of missing data to get an unbiased estimate of prediction accuracy during model building and assessment. While statistical modelling has a wealth of theoretical and practical information about missing data, there is less available for prediction modelling/machine learning. I want to build a prediction model and I am unsure how to handle the outcome observations Y during the model building/validation and testing procedure.
If I impute missing data among the predictors in the training data set using (for example) knn, I would develop the imputation model using X and Y in the training data set before I start identifying the best hyper parameters to develop a prediction model. I would use the imputation model to impute missing data in the validation data set to identify the best hyperparameters using cross-validation.
Would I include the outcome observations (Y) or would I remove the outcomes and treat them as missing to simulate a real-life situation? The same questions apply to the assessment of the final model in the hold-out test data set (or in the outer loop of nested cross-validation)?
I wonder if I can keep the outcome observations (Y) in the hyperparameter tuning step (validation data set) because my model performance will be only assessed in the hold-out/test data set, and including the outcome in the validation data set may improve the prediction model. However, I would not use the outcome in the final test data set (for internal validation)?
