# random forest variables importance with continuous and categorical variables and unbalanced output

I am a bit lost in the literature regarding the random forest importance. I am aware that there are different methods.

I have a binary output variables where elements labelled with 0 are much more than the ones labelled with 1.

5 variables are used as input. Some of them are continuous and some others are categorical.

Given a dataset of this type I am wondering what is the best method to asses variables importance with Random Forest and if this is available in any R or python library.

I know that the standard approach based the Gini impurity index is not suitable for this case due the presence of continuous and categorical input variables

• If there are only 5 input variables, why shouldn't you just remove each of them and check how this affected your results? Commented Oct 21, 2014 at 3:18
• I want to use a measure that is commonly used. Also your suggested strategy will influence the output according to how the tree is trained Commented Oct 21, 2014 at 9:08

Random forests for classification might use two kind of variable importance. See the original description of the RF here.

"I know that the standard approach based the Gini impurity index is not suitable for this case due the presence of continuos and categorical input variables"

This is plain wrong. The gini impurity is build using only the proportions of the target/dependent variable, when split by a test which involves either numerical or nominal independent variable. Note that the independent variable plays a role only for building the split test, the computation of gini index is based only on counts on dependent variable after split. Of course, the gini impurity index on each node is used further to compute gini importance.

I do not know if that would count, but my personal experiments revealed taht there are no big differences between gini variable importance and permutation value importance. And I usually prefered the former.

The second problem is the unbalance of the samples labeled with 1 and 0. I think this might play a role on variable importance, but to be honest I would verify if this is the case. Thus I would repeat many times various computations of variable importance with various sampels having different proportions varying gradually from 0.5 ratio to the actual ration. I expect that finding a stable variable importance no matter proportion to not be so unexpected.

[later edit]

It took me some time to compile the document provided by @Donbeo. I agree with the results from that paper and I hope that I would further experiment myself with that. The only think which I do not like about that study is that it does not state which number of trees were used and what would imply the variation of this parameter. The single note regarding that is that the number of trees affects the scaled version for permutation tests.

• I do not agree with you. The importance computed with the gini index is biased respect to predictors with more unique values. This has been stated here biomedcentral.com/1471-2105/8/25 Commented Oct 23, 2014 at 16:03
• First of all thank you for your link, I did not knew about it. It took me some time to parse it, but now I agree with the experimental setup and with the results. What do you think is better: delete may anser? or leave it like it is? I would not delete it for further reperence. PS: do you know about any further investigations regarding that? Commented Nov 5, 2014 at 13:15
• I think you should keep your answer maybe with a warning title or something like that. I think that these are the two main paper to consider. biomedcentral.com/1471-2105/9/307 biomedcentral.com/1471-2105/8/25 I had not the time to go deeply trough each paper but the problem and the solution seems to be pretty intuitive to me. Commented Nov 6, 2014 at 14:55

My current solution is a bootstrap on the samples such that the classes are balanced and the forest is trained using unbiased tree from the party package

sim = 100  #sim is the number of simulation. The larger the better and also the slower
s = 50 # s people with output 1 and s with output 0 are used in each iteration

x1 = x[x$asthma3==1,] x0 = x[x$asthma3==0,]
VI = data.frame(matrix(0, sim, ncol(x)-1))
cont = cforest_unbiased(ntree=1000, mtry=2)

for(i in 1:sim)
{

id1 = sample(x = nrow(x1), size = s, replace = F)
z1 = x1[id1,]
id0 = sample(x=nrow(x0), size = s, replace = F)
z0 = x0[id0,]
z = rbind(z0, z1)
#reasign value to label
cf = cforest(asthma3~., data=z, controls = cont)
vi = varimp(object = cf)
VI[i,] = as.numeric(vi)
}

names(VI) = names(x[,1:ncol(x)-1])
b=boxplot(VI, outline=F)

In general, for deriving feature importance in datasets that include a mixture of categorical & continuous data I would consider alternative methods such as permutation_importance from sklearn (I'm sure there is an equivalent in R)

There are a number of interesting examples and discussions on the sklearn website such as this one here.