1
$\begingroup$

Data:

  1. My training set consists of ~450k obs and 26 variables, out of which 1 is an ordinal factor (order_month, 12 levels) and the rest is numerical. Moreover, some of my predictors are highly correlated (Pearson's > 0.5), which is expected and sometimes intentional.

  2. I trained a good RF model (ROC 0.96, ~90% specificity & ~90% sensitivity when tested on unseen data) and I derived Variable importance for it using varImp().

Problem:

  1. Now, I know that having factorial and / or correlated predictors could result in biased variable selection in RF models, to which conditional inference trees (CIT) should be a remedy.

  2. I tried to compare results of randomForest::varImp() with party::varimp(myforest, conditional=TRUE) however I get an error in the latter:

Error in model.matrix.default(as.formula(f), data = blocks) : term 1 would require 9e+12 columns,

suggesting that I have either too many variables or variables with too many levels.

  1. On top of that it takes over 24hrs to run party::cforest() and running predict() on CIT model crashes my computer altogether. Given the above, I have a couple of

Questions:

1) how likely is it that I may completely misclassify the top VarImps in RF model. Is running CIT here absolutely necessary?

2) what are alternative ways of obtaining conditional variable importance if not through party::varimp()?

Big thanks for your suggestions! Kasia

$\endgroup$

closed as off-topic by mkt - Reinstate Monica, kjetil b halvorsen, Michael R. Chernick, Reinstate Monica, mdewey May 25 at 14:59

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – mkt - Reinstate Monica, kjetil b halvorsen, Michael R. Chernick, Reinstate Monica, mdewey
If this question can be reworded to fit the rules in the help center, please edit the question.

0
$\begingroup$

1) how likely is it that I may completely misclassify the top VarImps in RF model. Is running CIT here absolutely necessary?

Unlikely, because you have good results from random forest, unless you biase the probabilities obtained in the predict function. Random forest is an independent model so I don't see any absolute reason to run CIT.

2) what are alternative ways of obtaining conditional variable importance if not through party::varimp()?

There are others like: extendedForest and also this set of slides should give you more information.

Note

Please note that correlated predictors could result in biased variable selection in RF is not entirely true.: Ref: Why is multicollinearity not checked in modern statistics/machine learning

$\endgroup$
  • 1
    $\begingroup$ thanks, @discipulus, that's reassuring. However, even though I obtained good results from RF model, I'm not sure if the Variable Importance that it throws is correct, due to the issues described in the post. I know this won't affect the predictions, but I need to present the results to lay audiences and want to make sure that I explain the predictive power of the variables correctly $\endgroup$ – Kasia Kulma Feb 28 '17 at 9:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.