2
$\begingroup$

There is a quite old yet very good question about the proper way for using rfImpute but to me the question raised by Doug7 (whether the target variable y gets used for the imputation of the features Xi and whether that would be harmful later when trying to fit models to the new imputed data set) has not really been answered.

The accepted answer by RNB points towards using mice or missForest R packages for imputation instead of randomForest's rfImpute but the differences are that these packages

  1. build individual imputation models for each variable Xi while rfImpute builds a single model for the whole dataset
  2. you can choose to ex- or include the target variable y as a variable for all these individual imputation models in mice and missForest but you must include y in rfImpute

So I would like to raise the question about whether or not to use y during impute again but more generally and not limited to the use of rfImpute. The only study I could find about this is a 2017 study which focuses more on whether or not to also impute missing values of y but it also has some experiments with the potential to shed light on the question of whether or not to make use of y during imputation. Since I think there is an error for item #3 in the table listing the experiments they made let me list the first 3 experiments relevant to my question again here as I understand it:

  1. complete case analysis as a reference (no multiple imputation [mi])
  2. no imputation of missing y, y not used in mi model
  3. no imputation of missing y, y used in mi model

So in a nut shell, the question is: can you use strategy #3 from above (which would allow you using the computationally much cheaper rfImpute) or should you never make use of y in an imputation model (which would force you to go with something a lot more expensive like mice or missForest) and go with #2 instead when your concern is a model fit on the imputed data set?

My gut feeling told me not to use #3 but the study seems to show that when it comes to bias #3 clearly outperforms #2 and when it comes to error at least for larger data sets the same seems to be true. But this refers to the quality of the impute, not to the impact onto models fit on the new imputed data set!

What are your thoughts on this?

Thanks, Mark

$\endgroup$
5
  • $\begingroup$ What are you trying to do? What you should do in missing data imputation actually depends on your setting. E.g. are you trying to answer causal questions? Are you trying to build a prediction model? $\endgroup$
    – Björn
    Apr 1, 2023 at 22:19
  • $\begingroup$ I'm trying to build a model from which to derive feature importance. Predictions of this model play only a minor role. $\endgroup$
    – MarkH
    Apr 2, 2023 at 9:51
  • $\begingroup$ Feature importance for what purpose? Inferential interpretation? Feature selection for prediction modeling? Something else? $\endgroup$
    – Björn
    Apr 2, 2023 at 10:01
  • $\begingroup$ Feature importance for inferential interpretation (root cause analysis) $\endgroup$
    – MarkH
    Apr 2, 2023 at 10:10
  • $\begingroup$ You can't interpret feature importances causally. For one thing, the importances will be specific to the random forest model: it's the features that the RF uses to make its predictions. The true root cause of an event is not determined by what model you decide to use to analyze the data. $\endgroup$
    – dipetkov
    Apr 3, 2023 at 11:53

1 Answer 1

2
$\begingroup$

It's seldom good to argue "always" or "never" in response to this type of question. Much depends on the details of the data, your understanding of the subject matter, and what tradeoffs you are willing to make. What follows are some links to guidance that you can apply as needed.

Stef van Buuren's Flexible Imputation of Missing Data (FIMD) is a reliable reference. In general, the book doesn't even distinguish "outcomes" from "predictors" in terms of imputation. All data are typically lumped together into a single matrix with indicators of which elements are missing in the original data.

The missing-at-random (MAR) assumption underlying multiple imputation is that missingness can depend on observed but not unobserved information. Thus it makes sense to use as much of the observed information as reasonable in multiple imputation. I don't see any reason completely to ignore whatever information the observed $Y$ values might provide for imputation, as in your Strategy 2. You might choose for some reason not to, based on your understanding of the subject matter, but that should be a conscious choice.

Even if you choose not to use the outcome $Y$ in the imputation, it's still generally important to perform multiple imputations, fit models on each of the imputed data sets, and then combine the results as explained in Section 5. A quote from Rubin at the start of Chapter 2 puts it simply:

Imputing one value for a missing datum cannot be correct in general, because we don’t know what value to impute with certainty (if we did, it wouldn’t be missing).

As the page you link notes, confidence intervals and the like won't be correct if you only use a single imputation.

Section 2.7 of FIMD discusses some potential exceptions relating to this question.

First, your Strategy 3 never lets you use cases with a missing outcome value $Y$. That's OK if the only missing data are in $Y$. But that's not the situation that you are describing. Otherwise you run into the problems with complete-case analysis discussed in that Section and elsewhere:

The efficiency of complete-case analysis declines if $X$ contains missing values, which may result in inflated type II error rates.

Second, complete-case analysis is OK, in terms of bias, if the probability of missingness is independent of the outcome $Y$ in a regression model against predictors $X$:

The first special case occurs if the probability to be missing does not depend on $Y$. Under the assumption that the complete-data model is correct, the regression coefficients are free of bias...

That might argue for Strategy 1 if you can make that assumption about missingness! In that context, I think about single imputation with rfImpute() as extending complete-case analysis to cases with some missing $X$ values but observed $Y$ values that might otherwise be omitted. Insofar as the $Y$ values contain information about the values of the missing $X$ entries in the data matrix, that should tend to increase power (although it will again introduce problems with things like confidence intervals).

$\endgroup$
6
  • 1
    $\begingroup$ Strategy 3 seems to correspond roughly to von Hippel's suggestion to use $Y$ during imputation but delete all cases where $Y$ is missing during the analysis. According to Kontopantelis et al., as long as $Y$ is used during imputation, it doesn't seem to matter much. $\endgroup$ Apr 1, 2023 at 20:16
  • $\begingroup$ Thanks for this very detailed answer! The thing which makes me nervous with #3 is that during imputation you are building a model which e.g. imputes missing values in X1 as f(y, X2, ...., Xn). Based on this data set I'd then build a model for y as f(X1, ... Xn) where some of the X1 values partially depend on y. Just feels like a circular reference error. Is there no risk that based on y certain X1 values are imputed and this leads to incorrectly finding a high feature importance for X1 in the final model? $\endgroup$
    – MarkH
    Apr 2, 2023 at 10:22
  • $\begingroup$ @MarkH that's perhaps a reason why multiple imputation is often preferred. You then use a probabilistic imputation model that returns a set of imputations with different X1 values. The standard errors associated with X1 are thus larger after results are pooled by Rubin's rules, but the point estimates are unbiased if a proper imputation method is used. Sections 2.3 and those following of FIMD describe when and why multiple imputation works and the importance of multiple imputation with a proper model. $\endgroup$
    – EdM
    Apr 2, 2023 at 14:27
  • $\begingroup$ @MarkH the von Hippel reference provided by COOLSerdash recommends using all data for the multiple imputation, including the available outcome values and cases with missing outcome values. Thus there will be imputation of missing outcome values $Y$ during the process. After multiple imputation, you remove cases whose $Y$ values were imputed before you fit the outcome models and pool results. That's another way to deal with your fear about circularity. $\endgroup$
    – EdM
    Apr 2, 2023 at 14:54
  • $\begingroup$ Thanks! I'm thinking about running several rfImputes w/ different seeds as opposed to a proper multiple impute, von Hippel strategy is already part of the flow. Maybe this is good enough for feature importance and should be much quicker for this ad-hoc tool... $\endgroup$
    – MarkH
    Apr 2, 2023 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.