Background: I have a set of ~100 features (input) that predict 25 variables (output). My input variables are integers in {1,2,3,4,5,6,7}, my output is continuous. I have ~100K data rows available.

I would like to build a random forest where each tree can only split on 10 variables, that it can freely choose out of 100. But it can split as many times on that variable as it wants. In SKLearn:

  • max_features is a randomly assigned set of 10 features, but I want the tree to choose the 10 features it deems best, out of all 100 features.
  • max_depth and max_leaf_nodes counts multiple splits on a variable just the same as splits on different variables.
  • I don't see any other relevant parameters.

So how can I do this?

Background: I am aware that trees will often pick the same features. That is exactly what I aim for. I would like to see which features are the most/least necessary for prediction, so that I can make a regression method that predicts my output almost as good on ~80 of these features instead. The ones with the lowest feature importance in the resulting forest can be safely removed. If I don't do the feature limitation as described above, the feature importances are all a bit the same, since the features are very correlated.

For example, if feature A and B are both super important, but almost identical and A is a bit more important than B, then B can be left out, so with this approach, A will have a high feature importance and B will have a low feature importance in my approach. Where with plain RFs and max_features=10, both would have a very high feature importance.

I'm currently limiting the number of leaf nodes and it works fine, but it does mean I'm not fully using the power of each variable, since that limits the number of binary splits. And once you do one split on a variable, the other splits on that variable should be allowed for free.

  • $\begingroup$ Little confused. First, is this all just for feature selection? Second, you want the tree to pick 10 features, and want there to be some overlap between those choices (but presumably not all), but also want the trees to pick the 10 features they think are "best" -- is "best" random in some way? Why won't the 10 features chosen always be the identical "best" ones? Side point, it seems like you might want to use a single decision tree to choose ten good features, because it won't have the problem with redundant features (perhaps as a step in a larger process). $\endgroup$ – one_observation Jan 12 '16 at 18:47

I guess what your looking for a way to quantify variable redundancy or the opposite complimentarity('new word'). A and B can either be replacable or complementary, hence removing A will make B less usefull/important as feature. E.g. knowing the lab who produced a doping analysis (B) may be be pretty useless without knowing the result(A). The problem is that C, D could be redundant also or complimentary with A and B. Simply rerunning analysis without feature A, may not strongly indicate B is redundant as C and D was also redundant.

I would implement your idea as an ensemble of random forest models, where each forest has a random variable restriction. Write a wrapper around your favorite random forest algorithm and train multiple models. Go invent some nice statistics relating variable selection to OOB-CV performance and variable importance. The procedure would be pretty experimental, but it could be fun.

I have experimented with 2way-variable importance: To measure the change of variable importance if another variable were permuted also. If variable importance increase, the variable pair were redundant. If variable importance decrease the to variables were complimentary. Variable complimentatity is observed for variables with a dominant interaction effect, otherwise not. If no change, then perhaps no redundancy or no complimentarity. Perhaps the redundancy were masked by other variables. I sketched the details of this implementation here

Recursively dropping the remaining variables with 5% lowest variable importance and retrain does in practice what your looking for in a simple way. You need a external CV to evaluate if your model performance has increased.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.