In the Random Forest paper they describe a nice way of measuring a variable importance - take your validation data, measure error rate, permute the variable and re-measure error rate.

Question - why is that method specific to Random Forests? I understand that in other classifiers (SVM, LR, etc.) we don't have the concept of OOB, but we certainly can use a regular train-validation split.

What am I missing here? Why isn't this method a common practice?

  • 4
    $\begingroup$ Fair question. As I see it, any supervised method can be bagged and VI can be comouted. It may be costly to train 50 svm models. For RF the VI comes with little extra cpu time. $\endgroup$ Commented Sep 15, 2015 at 23:17
  • 1
    $\begingroup$ @SorenH.Welling - I think your comment hints what I personally was missing. I thought of variable importance as some scalar you can fairly estimate using simple 3-fold X-validation mean. However, Breiman envisions it as a distribution - only after 50 or so X-validations you can have such a distribution, with a mean importance for each feature, and what's more important - a p-value for it. This is something you'll get easily only from Random Forests. Certainly not for SVM or some other fancy supervised learner. $\endgroup$
    – ihadanny
    Commented Sep 20, 2015 at 20:51

2 Answers 2


Any bagged learner can produce an analogue of Random Forests importance metric.

You can't get this kind of feature importance in a common cross-validation scheme, where all the features are used all the time.


Random Forrest and other techniques that incorporate bagging are using the fact that the bootstrap sample that is drawn for the current tree excludes some data points, the so-called Out-Of-Bag samples (OOB). Since these samples are not used to build the current tree, they can be used to evaluate it without the risk of overfitting. With other supervised learning techniques that usually do not suffer from instability as much as decision trees (e.g. SVM), you usually do not draw bootstrap samples and thus you can not estimate variable importance in this way.

However, the approach of training a model with different subsets of variables and evaluate their performance using k-fold cross-validation is also perfectly valid and called Wrapper approach in the literature. For instance, a popular feature selection technique with SVM is recursive feature elimination (see https://pdfs.semanticscholar.org/fb6b/4b57f431a0cfbb83bb2af8beab4ee694e94c.pdf)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.