3
$\begingroup$

For a classification project we are using the randomForest package in R, which wraps the Breiman Fortran random forest implementation, to assess the importance of each of our features. I would like to calculate p-values for each feature's importance statistics as described in the random forest documentation provided by Breiman.

... therefore we compute standard errors in the classical way, divide the raw score by its standard error to get a z-score, ands assign a significance level to the z-score assuming normality.

The R RandomForest package provides the mean decrease importance (MDI) metric for each of the classes and overall (both classes combined) in addition to the standard deviation for the decrease importance of each class and overall.

I don't understand how these values can be used to obtain a significance level for the variable importance as, while the mean and standard deviation will allow the construction of the a normal distribution, there is no "observation" for the z-score calculation. Can someone clarify how to do this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It is called z-score mainly because it is mean/sd, but in practice it is useless for hypothesis testing -- in some cases you can get most important attribute with z~$10^{-3}$ or on the other side all z-scores way larger than this mystical 3.

The working (more-less) approach is for instance to compare attributes' importance to an importance of random dummy attributes added to the set. I have made a package, Boruta, that implements such idea.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Yes, I know what a z-score is and I understand how to calculate the significance based on permuted variable values, what I don't understand is how to do this from the information provided by the randomForest package in R: importance mean and importance standard deviation for each feature. $\endgroup$
    – Nixuz
    Oct 25, 2011 at 21:25
  • 1
    $\begingroup$ Well, you can't, that's the point. You must manually add this datum attributes, train the forest, get importance and do the test on original versus permuted. $\endgroup$
    – user88
    Oct 25, 2011 at 22:38
  • $\begingroup$ There is no way to exploit the fact that the distribution given is the difference between the permuted and actual? $\endgroup$
    – Nixuz
    Oct 26, 2011 at 0:19
  • $\begingroup$ The problem is that this gives not enough power to do the test; the attributes are not independent, the trees may also have some correlation, $N_\text{tree}\left<\text{depth}\right>/N_\text{attr}$ is not enough, etc. In short, it does not work. $\endgroup$
    – user88
    Oct 26, 2011 at 15:57
  • $\begingroup$ What about a permuted measure of variable importance? I.e., re-run RFs with permuted class labels, say 1,000 times, and get the approximate $p$-values for the original importance measure? $\endgroup$
    – chl
    Oct 26, 2011 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.