How to calculate variable importance taking correlated predictors into account?

Question

This is the problem I am facing right now:

I have a dataset with 100.000 samples and 20 predictors. The predictors are correlated with each other due to their nature. I've run two different random forest models with a sample of the dataset (see why I used a subsample in the explanation below):

1. The first random forest was from the party package
2. The second random forest was from the randomForest package

My objective, aside from predicting a certain variable with the model, is to find out the importance of the predictors.

According to the test I ran with a subsample of the entire dataset, I got contraddictory answers on variable importance using party and randomForest.

Party, when using the function:

varimp(cf_fit,conditional=TRUE)

with the argument conditional set to TRUE, seems to output an answer that "makes more sense" (based on my knowledge of the subject) compared to the answer of the randomForest's

importance()

function.

By reading a bit deeper into the literature, I found out that this is common when using correlated predictors. Since the party package is partially doing a better job than the alternatives, I would like to use it, however, when feeding the whole dataset, the function varimp(cf_fit,conditional=TRUE) crashes the program. Again, I read that this is quite common or, at least, it is common for this function to be really slow, which it definitely is, even when using only a subsample.

Finally, my question:

How can I calculate the variable importance accounting for variable correlation? (ie: how can I do what the function varimp(cf_fit,conditional=TRUE) is doing so slowly at best, and crashing at worst?)

My proposed answers:

1) I simply don't do it this way and try to find the best predictors by running different times the random forest with different subset of predictors (possibly those with the least correlation among themselves), then I choose the independent predictors yielding the best results and only then I calculate variable importance.

2) Use a faster machine and try using party's varimp(cf_fit,conditional=TRUE) function anyway. I'm not sure this is a hardware problem... speculation time... is it maybe related to the way this function works?

Thanks to everyone in advance!

Answer 1

If you use the random forest variable importance, you should make sure that the hyper-parameter are tuned. Results will not be meaningful if you perform variable importance with a model that over/underfits. Assuming you have done this there are two other options apart from the rf variable importance that I can think of to perform this feature selection task:

1. Wrapper approach: train a model with the full set of variables and step-wise remove variables with the lowest variable importance score. Stop until the prediction error does not decrease anymore (backwards feature elimination). This can be also done the other way around, i.e. start with one variable and step-wise add a the one with the highest importance score.

2. Filter Approach: There exists many different heuristics to perform feature selection called filter methods. Some of them called multivariate filters also take into account correlation among the variables and remove redundant variables, e.g. Minimum redundancy Maximum Relevance (mRMR) or Conditional Mutual Information Criterium (CMIM).

The first option treats the number of variables essentially as a hyper-parameter. The second option will be faster, but often less reliable. Performing PCA would also be an option for highly correlated variables assuming that you just care about the prediction results and not about interpretation.