random forest variables importance with continuous and categorical variables and unbalanced output

Question

I am a bit lost in the literature regarding the random forest importance. I am aware that there are different methods.

I have a binary output variables where elements labelled with 0 are much more than the ones labelled with 1.

5 variables are used as input. Some of them are continuous and some others are categorical.

Given a dataset of this type I am wondering what is the best method to asses variables importance with Random Forest and if this is available in any R or python library.

I know that the standard approach based the Gini impurity index is not suitable for this case due the presence of continuous and categorical input variables

Answer 1

Random forests for classification might use two kind of variable importance. See the original description of the RF here.

"I know that the standard approach based the Gini impurity index is not suitable for this case due the presence of continuos and categorical input variables"

This is plain wrong. The gini impurity is build using only the proportions of the target/dependent variable, when split by a test which involves either numerical or nominal independent variable. Note that the independent variable plays a role only for building the split test, the computation of gini index is based only on counts on dependent variable after split. Of course, the gini impurity index on each node is used further to compute gini importance.

I do not know if that would count, but my personal experiments revealed taht there are no big differences between gini variable importance and permutation value importance. And I usually prefered the former.

The second problem is the unbalance of the samples labeled with 1 and 0. I think this might play a role on variable importance, but to be honest I would verify if this is the case. Thus I would repeat many times various computations of variable importance with various sampels having different proportions varying gradually from 0.5 ratio to the actual ration. I expect that finding a stable variable importance no matter proportion to not be so unexpected.

[later edit]

It took me some time to compile the document provided by @Donbeo. I agree with the results from that paper and I hope that I would further experiment myself with that. The only think which I do not like about that study is that it does not state which number of trees were used and what would imply the variation of this parameter. The single note regarding that is that the number of trees affects the scaled version for permutation tests.

Answer 2

My current solution is a bootstrap on the samples such that the classes are balanced and the forest is trained using unbiased tree from the party package

sim = 100  #sim is the number of simulation. The larger the better and also the slower
s = 50 # s people with output 1 and s with output 0 are used in each iteration




    x1 = x[x$asthma3==1,]
        x0 = x[x$asthma3==0,]
    VI = data.frame(matrix(0, sim, ncol(x)-1))
    cont = cforest_unbiased(ntree=1000, mtry=2)

    for(i in 1:sim)
    {

      id1 = sample(x = nrow(x1), size = s, replace = F)
      z1 = x1[id1,]
      id0 = sample(x=nrow(x0), size = s, replace = F)
      z0 = x0[id0,]
      z = rbind(z0, z1)
      #reasign value to label
      cf = cforest(asthma3~., data=z, controls = cont)
      vi = varimp(object = cf)
      VI[i,] = as.numeric(vi)
    }

    names(VI) = names(x[,1:ncol(x)-1])
    b=boxplot(VI, outline=F)

Answer 3

In general, for deriving feature importance in datasets that include a mixture of categorical & continuous data I would consider alternative methods such as permutation_importance from sklearn (I'm sure there is an equivalent in R)

There are a number of interesting examples and discussions on the sklearn website such as this one here.