I'm using the
mutualinformation function from the "infotheo" package in R in order to select my predictors for my Random Forest model. In addition I use RF's
importance for the same reason.
I have a predictor which comes third in importance by those two methods, but each time I include it, the accuracy of the model is going down. It has a 0.5 correlation with one other, lower graded, predictor, but removing it didn't help. i tried different combinations with that predictor but nothing gave me better results than without it.
How can this situation be? I assume every variable which scores high in explaining the target values should improve the model's accuracy (unless it is highly correlated with others, of course)