I'm working on a project with medical data where some of it is missing. We decided to impute the data using MICE and I found enough literature about how to choose $m$ (the number of imputations) and $t$ (the number of iterations), which variables to keep for imputation and so on.
The task is to build a classification algorithm on that data.
However, it is surprisingly hard to find anything about Multiple Imputation in a Machine Learning context (where I'm not interested in means, variances, test statistics, confidence intervals, and so on but point estimates). I read Stef van Buuren's great book and found the recommended workflow: https://stefvanbuuren.name/fimd/workflow.html I checked Rubin's "Multiple Imputation for Nonresponse in Surveys" for how the pooling should be done, which will might for regression coefficients, but only if the same features are selected on each dataset and if linear/logistic regression is used in the first place.
I can conceive different ways to handle the $m$ differently imputed datasets:
- Use some valid way of statistical "pooling" (how exactly)
- Build $m$ different models and average their predictions (and cross-validate each model on each imputed dataset, and average the CV results)
- Stack the data into one dataset (cross-validate by performing the same CV split on each imputation so that no test instances leak into the training set and then combine the imputations)
I implemented the 2nd and 3rd alternative and notice that the classification metrics differ (slightly). What is the correct way to handle the different $m$ datasets, so that in the end we get realistic performance estimates and one model to put into production?
Van Buuren reads:
If the scientific interest is restricted to point estimates and if the complete-data model is linear, this analysis of the stacked imputed data will yield unbiased estimates.
But what if my models are not linear? What if I use tree-based algorithms?
I'm glad for any answer that can help me find and justify my approach.