1
$\begingroup$

I'm puzzled about why a dependent variable with the weakest correlation to the outcome variable emerges as the most important factor when I run my random Forest on the same dataset. It beats out factors with much stronger correlations with the outcome variable.

Now I'm not sure whether to trust my random Forest. All the predictor variables were numeric and the outcome variable was a factor variable with two levels.

Do you have insight about 1) why this happens and 2) what I should do next to assess which variables are in fact most predictive?

The RF code I used was:

rf_mod <- randomForest(Stable ~ ADF + EstAnnualVal + PR_Score + stdv_demand_intervals,
                       data=var1, ntree=501, importance=TRUE)
$\endgroup$

migrated from stackoverflow.com May 29 '16 at 23:55

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ I'm guessing you have interaction effects between one or more variables, such that there might be almost no correlation of a to b directly, but there might be between a and b at each level of c $\endgroup$ – thelatemail May 29 '16 at 22:41
  • $\begingroup$ You haven't been very specific about how you've calculated correlation and importance. In any case there's a general issue: are you interested in how predictive each variable is on its own, or in how much predictiveness it adds on top of the other variables? Those are quite different questions. $\endgroup$ – Scortchi May 30 '16 at 14:44
2
$\begingroup$

I am not claiming that this is a case for your data (as this requires analysis of the data), but here is a general, relatively important, issue with depending on correlation:

Correlation of the variable with the dependent one has the same assumption as Naive Bayes - independence of the variables. Thus, the only thing you are able to test this way is "in the absence of all other features, how well would I describe my dependent variable, if I fit a monotonic model to this one, selected feature". These are two, extremely strong, assumptions. Consequently you completely omit any, even a bit complex, relation. For example if you have a numeric variable, such that every time it is odd, your true label is 1, and every time it is even it is 0 - your correlation test will say "there is nothing interesting in this variable", while you have full knowledge about dependent variable, as it is y = x % 2. Furthermore, even if the relation is monotonic and simple, but involves multiple factors - you will miss it too!

Random Forest, on the other hand, does not make this kind of assumptions, it builds a complex, nonlinear model, and assess variables importance according to degree to which it is used in its internal decision process. Consequently - it is incomparably better method of assessing the importance, than the correlation test.

$\endgroup$
2
$\begingroup$

Recall, random forests (RF) assigns split values for each feature, which is not the same as correlation. Fundamentally, RF, is a decision tree type classifier which identifies the best value of each feature to split on.

Correlation and RF is very different -- and there should be little expectation for similar results.

Within RF at run-time, each feature is randomly selected for splitting a parent node (in the tree) into two daughter nodes, and the value of the feature where the split occurs depends on the objects in the parent node which need to be partitioned. Within these objects, there can be any combination of classes (dependent variable values), because of the previous object split histories of parent nodes in the splits above. In other words, when a feature is randomly selected for making a split, only parts of the feature's values are used because only part of the objects are in the parent node. You'll rarely have all the objects that you ran correlation on within a parent node. Even when you start RF, you're doing CV training/testing, so you'll only have objects for all of the training folds in the first parent node. The training set is also a bootstrap, so there can be multiple copies of the same objects in any node.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.