2
$\begingroup$

I was wondering what imputation method you would recommend for data to be fed into a random forest model for a classification problem.

If you google for "imputation for random forests", you get a lot of results about imputation with/by random forests, but next to nothing for imputation for random forests.

My understanding is that for random forest models, the distribution of each predictor is important, so using a simple method like class-mean imputation would be incorrect, because it distorts the distribution of the predictors.

So, what imputation method(s) would you recommend for random forests?

In my specific case, I'm building this model for 2 purposes, but please feel free to answer the question in a more general way. My 2 purposes are:

  • predicting the class of future, unlabelled samples with no missing values
  • determine what the most important features are, and build a story based on that

And if I need to give more detail, please let me know. (I feel if I include a lot of detail to begin with it'll be distracting)

Thanks very much!

$\endgroup$
  • $\begingroup$ What are you going to do with that random forest? $\endgroup$ – Michael M Jun 16 at 13:02
  • $\begingroup$ thanks for your comment - i've added that to the question $\endgroup$ – Dave Jun 16 at 13:15
  • 1
    $\begingroup$ Thanks. At least for the first question you will need a very good test and cross-validation strategy that includes the imputation part. Otherwise, there is no reliable way to guesstimate accuracy of the model. $\endgroup$ – Michael M Jun 16 at 13:24
0
$\begingroup$

You shouldn't as what imputation method is best for your model, but rather how will the imputation method affect your model.

I think the best approach is to compare imputation methods in your cross validation. Thus, the imputation essentially becomes a sort of hyperparameter.

Anyway, the answer is highly context dependent, and can really only be answered via cross validation. What works for one model/problem/dataset may not work for the next.

$\endgroup$
  • $\begingroup$ Thanks for your answer! If I understand correctly, you're saying I should simply try several methods and then see which one ends up giving the highest cross-validated accuracy. Is that correct? $\endgroup$ – Dave Jun 15 at 19:43
  • 1
    $\begingroup$ Yea, I think that sounds like the right approach. Most common ML libraries can make the imputer a part of the pipeline, making it easy to cross validate the approach. $\endgroup$ – Demetri Pananos Jun 15 at 19:44
0
$\begingroup$

I would say, it strongly depends 'where' the data are missing. Only in your training set? or in the test set too? do you expect missing data when actually predicting on new data?

If you have missing data in your test set as well, and you do some sophisticated single-imputations based on a regression or something like that, then you work with the 'most-likely' values in both sets. This will lower your testing-error as you artificially take variance out of the data

$\endgroup$
  • $\begingroup$ Thanks for you answer. Let me give you some more detail. I don't know if I'm going to be splitting my dataset into training and testing, because I only have 140 samples, and many predictors. There are 95 predictors, of which I'm only using 60 because there are 35 predictors that have more than 30% missing values, so I'm getting rid of those. Of the 60 predictors, I'm hoping to select a maximum of 7 for this prediction/classification problem. Within each predictor, missing values seem to be at random and are not related to the classes. Future data will not come with missing values. $\endgroup$ – Dave Jun 16 at 8:34
0
$\begingroup$

Your point "the distribution of each predictor is important" is always the case. Your conclusion from this fact "so using a simple method like class-mean imputation would be incorrect, because it distorts the distribution of the predictors" is not so clear to me. Why can't you use class mean imputation as a first strategy?

$\endgroup$
  • 1
    $\begingroup$ The reason I feel so apprehensive abourt using class mean imputation is that it reduces the internal variance of each class, and I'm worried that that is artificially making each class more homogeneous internally, and more different from the other classes. That would then make the classification results artificially better than they would be if I was using a method that didn't distort the distributions. Is that wrong? $\endgroup$ – Dave Jun 16 at 9:57
  • $\begingroup$ It does reudce internal variance of each class and you are right that you might get a biased data set. Class mean imputation is certainly not an optimal imputation strategy. What I meant to say was that as a first step you could try class mean imputation. $\endgroup$ – Philipp-Packmohr Jun 16 at 10:21
  • $\begingroup$ Of course. I've tried that already. And it results in something like 85% internal cross-validation accuracy of my RF classification model. That sounds really good, but I'm worried that I'm biasing my data with the imputation. So I want to use a couple other imputation methods and see if the model is picking the same predictors and if the accuracy is similar. So I'm wondering what other methods I should try, apart from class mean. $\endgroup$ – Dave Jun 16 at 10:30
0
$\begingroup$

okay, so omitting more data, will probably not be possible

In my opinion, there is one common mistake with imputations: people tend to use those imputation that lead to a better model fit. But of course the model is not getting better, they imputed just the fitting values. A way to generally lower that risk are multiple imputations...but it's not possible to combine this method with randomforests (yet), at least as far as I know.

My understanding is that for random forest models, the distribution of each predictor is important, so using a simple method like class-mean imputation would be incorrect, because it distorts the distribution of the predictors.

I don't think that is correct. I would describe randomforests as 'very forgiving' to imputing a 'mean'. Even if that 'mean' would be terribly wrong, the 'random' part of an 'randomForest' deals with that easily

The reason I feel so apprehensive abourt using class mean imputation is that it reduces the internal variance of each class, and I'm worried that that is artificially making each class more homogeneous internally, and more different from the other classes. That would then make the classification results artificially better than they would be if I was using a method that didn't distort the distributions. Is that wrong?

It's completely the other way around--let's say you have two observations. X1 is 100kg and height unknown. X2 is 50 kilo and height unknown. You are using a regression-prediction as the imputation. The most likely height for X1 is 1.90m and for X2 1.50m. So your Residuals have a very low variance and might not display the reality, where people do not always have the most likely height for their weight. also , if those two observations get splittet up in training and test, it will be more easy for them to "find the patterns" as their values were created in the same data-generating process.

instead if you are using the mean, your error's variance will usuallly get higher

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.