Suppose I am fitting a Random Forest model with A-F as my predictors and Y as my response variable.

I then calculate variable importance using the permutation method.

Why is it possible for variable importance be smaller for A compared to B when A has a much higher correlation with the response Y?

  • $\begingroup$ Plot your data - it might help $\endgroup$ – probabilityislogic Jun 8 '18 at 13:25
  • $\begingroup$ If correlation matters--as measured by the Pearson correlation coefficient--then you should be performing ordinary least squares regression rather than using a random forest. $\endgroup$ – whuber Jun 8 '18 at 13:31

Absolutely. Correlation gives the linear relationship between two variables. It will (typically) not be able to find non-linear relationships. Here's an (over?) simple example. Suppose that x = 1:1000 and that y = +1 500 times and -1 the next 500 times. cor(x,y) <1 (since y is non-linear). Otoh the simple tree with one split - is x<= 500 will perfectly classify y.

  • $\begingroup$ Thank you! This makes perfect sense and makes things very clear $\endgroup$ – Grint Jun 8 '18 at 13:57
  • $\begingroup$ Thks, I don't like my example though. I'm going to add a better one. $\endgroup$ – aginensky Jun 8 '18 at 13:59
  • 1
    $\begingroup$ So here's another example I like better, although it's not perfect. Predict x from abs(x) and sin(x). If the interval of x is symmetric around 0, then cor(x,abs(x)) = 0. Otoh, abs(x) will usually be a better predictor than sin(x). I don't have time to run a tree though :). $\endgroup$ – aginensky Jun 8 '18 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.