Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

For a classification project we are using the randomForest package in R, which wraps the Breiman Fortran random forest implementation, to assess the importance of each of our features. I would like to calculate p-values for each feature's importance statistics as described in the random forest documentation provided by Breiman.

... therefore we compute standard errors in the classical way, divide the raw score by its standard error to get a z-score, ands assign a significance level to the z-score assuming normality.

The R RandomForest package provides the mean decrease importance (MDI) metric for each of the classes and overall (both classes combined) in addition to the standard deviation for the decrease importance of each class and overall.

I don't understand how these values can be used to obtain a significance level for the variable importance as, while the mean and standard deviation will allow the construction of the a normal distribution, there is no "observation" for the z-score calculation. Can someone clarify how to do this?

share|improve this question
add comment

1 Answer

It is called z-score mainly because it is mean/sd, but in practice it is useless for hypothesis testing -- in some cases you can get most important attribute with z~$10^{-3}$ or on the other side all z-scores way larger than this mystical 3.

The working (more-less) approach is for instance to compare attributes' importance to an importance of random dummy attributes added to the set. I have made a package, Boruta, that implements such idea.

share|improve this answer
    
Yes, I know what a z-score is and I understand how to calculate the significance based on permuted variable values, what I don't understand is how to do this from the information provided by the randomForest package in R: importance mean and importance standard deviation for each feature. –  Nixuz Oct 25 '11 at 21:25
1  
Well, you can't, that's the point. You must manually add this datum attributes, train the forest, get importance and do the test on original versus permuted. –  mbq Oct 25 '11 at 22:38
    
There is no way to exploit the fact that the distribution given is the difference between the permuted and actual? –  Nixuz Oct 26 '11 at 0:19
    
The problem is that this gives not enough power to do the test; the attributes are not independent, the trees may also have some correlation, $N_\text{tree}\left<\text{depth}\right>/N_\text{attr}$ is not enough, etc. In short, it does not work. –  mbq Oct 26 '11 at 15:57
    
What about a permuted measure of variable importance? I.e., re-run RFs with permuted class labels, say 1,000 times, and get the approximate $p$-values for the original importance measure? –  chl Oct 26 '11 at 18:41
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.