I'm using the mutualinformation function from the "infotheo" package in R in order to select my predictors for my Random Forest model. In addition I use RF's importance for the same reason.

I have a predictor which comes third in importance by those two methods, but each time I include it, the accuracy of the model is going down. It has a 0.5 correlation with one other, lower graded, predictor, but removing it didn't help. i tried different combinations with that predictor but nothing gave me better results than without it.

How can this situation be? I assume every variable which scores high in explaining the target values should improve the model's accuracy (unless it is highly correlated with others, of course)

  • $\begingroup$ you are talking about of out of sample accuracy, right? $\endgroup$ – Aksakal Sep 19 '17 at 14:01
  • $\begingroup$ Indeed. And when I run RF+CV, the best model's accuracy goes down as well. $\endgroup$ – Riddle-Master Sep 19 '17 at 15:55
  • 1
    $\begingroup$ As a top of mind, the mutualinformation function would return you a plug-in value (estimate) by default. In case of small sample size, or huge number of levels in your cat.variables this value is overestimated. Second, removing variables from RF model will most often not contribute to increasing the model's accuracy, but it will make the model simpler (more robust). $\endgroup$ – Alexey Burnakov Sep 21 '17 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.