Handling "missing" data, when the missing variable is the most informative feature

Question

I'm working on a project developing a predictive model for whether or not an individual has a (rare) disease based on some non-invasive test results. The idea is that this could help patients avoid lengthy and invasive tests.

One of the non-invasive techniques is very predictive for this particular disease; if a patient has received this test and has a certain combination of results (the test returns several different results), it can be very predictive for this disease. However, only a small subset (10-20%) of patients in the dataset have received the test, as it is not offered at all facilities.

What would be an appropriate way of modeling these data? I could be wrong but my intuition is that imputation doesn't make sense here. Would a random forest or gradient-boosted model work best, since they can handle "missing" data well? A colleague suggested building two models, one with and one without the data in question, but I'm not sure how that would work.

Thank you!

Answer 1

In principle, multiple imputation might be applicable. The missing data might be considered "missing at random" in the technical sense explained in that reference, because your data seem to contain the reason for the missingness: some facilities provide it, others don't. Using all of your available data in a well-designed multiple imputation model could provide a way forward.

That might particularly be true in your situation, where you hypothesize that your (often missing) non-invasive test of interest adequately represents the results of invasive tests, whose results are presumably also available in your data. In that case, the results of the invasive tests might be used to get reasonably consistent imputations of the results of the non-invasive test.

That approach repeats the modeling on multiple data sets, each with imputations done probabilistically. You combine the results of the multiple models in a way that takes the uncertainty in imputation into account. In general, with so many missing data values, you might have very wide confidence intervals around your estimates for the association between the non-invasive test of interest and the disease status. If your hypothesis about the ability of the non-invasive test to capture information provided by other tests holds, however, then there might be little enough variability in its imputed values to provide reasonable estimates.

You say that random forests and gradient boosting handle missing data well. That can be true, but be sure to know which of several approaches are used in your implementation. See the discussion on this page. Imputation is one type of missing-data handling in those types of models, too.