Distribution of variable importance in r party package I have a dataset of 14558 rows and 250 variables. I am trying to solve a classification problem thanks to r party package and the cforest function (which corresponds to a Random Forest).
I would like to obtain the value of variables importance for each tree instead of only getting the mean of the variable importance over all trees (this is what the the varimp function does).
I have in mind to do a boxplot with those variables importance value in order to evaluate the confidence of the value I have. If someone has another idea to do it, please let me know ! :)
I was thinking that it would be possible since we see it in the following paper : https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-307 
I have search over this forum and the party package documentation and did not find the solution. I hope I am not opening an already existing topic !
Thanks for your help :)
Rémi
 A: To obtain a measure of uncertainty for the permutation-based variable importance, you can simply generate a large number of random permutations and then look at their distribution. This is what we have done in the figure from the manuscript you cite.
Note that this is quite different from looking at the variable importances of the individual trees which would likely be misleading. The individual trees can be quite weak classifiers and hence have misleading importances. So you should really assess the variability of the forest importances not the tree importances.
The varimp() implementation in the party package by default just uses a single random permutation of each variable. But it allows to set varimp(..., nperm = ...) to a larger number. However, in that case it only returns the mean of the nperm permutations. But if you want the full distribution you can also easily replicate(..., varimp(...)) the desired number of times.
For example, you can fit the forest from the ?varimp examples:
library("party")
set.seed(1)
cf <- cforest(score ~ ., data = readingSkills,
  control = cforest_unbiased(mtry = 2, ntree = 50))

And for each variable you can generate, say, 50 random permutations for each predictor and visualize these with a boxplot:
vi <- t(replicate(50, varimp(cf)))
boxplot(vi)


