1
$\begingroup$

Data:

  1. My training set consists of ~450k obs and 26 variables, out of which 1 is an ordinal factor (order_month, 12 levels) and the rest is numerical. Moreover, some of my predictors are highly correlated (Pearson's > 0.5), which is expected and sometimes intentional.

  2. I trained a good RF model (ROC 0.96, ~90% specificity & ~90% sensitivity when tested on unseen data) and I derived Variable importance for it using varImp().

Problem:

  1. Now, I know that having factorial and / or correlated predictors could result in biased variable selection in RF models, to which conditional inference trees (CIT) should be a remedy.

  2. I tried to compare results of randomForest::varImp() with party::varimp(myforest, conditional=TRUE) however I get an error in the latter:

Error in model.matrix.default(as.formula(f), data = blocks) : term 1 would require 9e+12 columns,

suggesting that I have either too many variables or variables with too many levels.

  1. On top of that it takes over 24hrs to run party::cforest() and running predict() on CIT model crashes my computer altogether. Given the above, I have a couple of

Questions:

1) how likely is it that I may completely misclassify the top VarImps in RF model. Is running CIT here absolutely necessary?

2) what are alternative ways of obtaining conditional variable importance if not through party::varimp()?

Big thanks for your suggestions! Kasia

$\endgroup$

1 Answer 1

0
$\begingroup$

1) how likely is it that I may completely misclassify the top VarImps in RF model. Is running CIT here absolutely necessary?

Unlikely, because you have good results from random forest, unless you biase the probabilities obtained in the predict function. Random forest is an independent model so I don't see any absolute reason to run CIT.

2) what are alternative ways of obtaining conditional variable importance if not through party::varimp()?

There are others like: extendedForest and also this set of slides should give you more information.

Note

Please note that correlated predictors could result in biased variable selection in RF is not entirely true.: Ref: Why is multicollinearity not checked in modern statistics/machine learning

$\endgroup$
1
  • 1
    $\begingroup$ thanks, @discipulus, that's reassuring. However, even though I obtained good results from RF model, I'm not sure if the Variable Importance that it throws is correct, due to the issues described in the post. I know this won't affect the predictions, but I need to present the results to lay audiences and want to make sure that I explain the predictive power of the variables correctly $\endgroup$ Commented Feb 28, 2017 at 9:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.