5
$\begingroup$

I cannot find out how the variable importance for classification problems is calculated in h2o. There is a Stackoverflow question asking the same, but the accepted answer does not help (keeps referring to "squared error" where I would expect "accuracy" or "gini impurity" being used; same for the linked paper in that SO thread).

Note that h2o seems to use a different methodology for calculating variable importance than the usual permutation approach, see the h2o documentation

How is variable importance calculated for DRF? Variable importance is determined by calculating the relative influence of each variable: whether that variable was selected during splitting in the tree building process and how much the squared error (over all trees) improved as a result.

So, I tried to figure out how h2o calculates variable importance myself. Here a simple single-tree example (using all data for training)

library(h2o)
data(iris)

h2o.init()
irisSimple=iris
irisSimple$Species=factor(ifelse(irisSimple$Species=="virginica",
                                 "virginica","other"))
mdl=h2o.randomForest(x=setdiff(colnames(irisSimple),"Species"),
                     y="Species",training_frame=as.h2o(irisSimple),
                     sample_rate=1.0,ntrees=1,seed=1)

We can look into the single tree via exporting to a POJO

pojo=capture.output(h2o.download_pojo(mdl))

Now extract and print the first split node

pojo[grepl("double pred = ",pojo)]
#double pred = (Double.isNaN(data[3]) || data[3 /* Petal.Width */] <1.75f ?

Calculate left (true) and right (false) data bins

lBin=irisSimple[irisSimple$Petal.Width<1.75,]
rBin=irisSimple[irisSimple$Petal.Width>=1.75,]

Finally calculate accuracy increase

rootCorrect=max(table(irisSimple$Species))
lCorrect=max(table(lBin$Species))
rCorrect=max(table(rBin$Species))
accIncrease=(lCorrect+rCorrect-rootCorrect)/nrow(iris)
accIncrease
#[1] 0.29333

and compare to the h2o result

h2o.varimp(mdl)
#Variable Importances: 
#      variable relative_importance scaled_importance percentage
#1  Petal.Width           28.585253          1.000000   0.857558
#2 Petal.Length            3.081414          0.107797   0.092442
#3  Sepal.Width            1.000000          0.034983   0.030000
#4 Sepal.Length            0.666667          0.023322   0.020000

Summing up sum(h2o.varimp(mdl)$relative_importance) gives 33.33 indicating that relative_importance refers to the accuracy increase (the naive model assigning "other" to all observations has 50 observations wrong; the decision tree gets all 150 observations right).

As you can see, my calculated accuracy increase of 0.29333 for the Petal.Width split point is larger than the h2o value of 0.28585.

So, I am wondering what numbers h2o is reporting...

BTW:

packageVersion("h2o")
#[1] ‘3.10.5.3’
$\endgroup$
  • $\begingroup$ I agree that the current explanation is not detailed enough. I've added a ticket to improve this: 0xdata.atlassian.net/browse/PUBDEV-5176 $\endgroup$ – Erin LeDell Dec 19 '17 at 4:09
  • $\begingroup$ @ErinLeDell Thx! It would be also nice if you guys could add documentation on which data the VarImp is based on. For instance, is it calculated on the validation data set if I supply a validation frame? Does it calculate it on OOB data of the training_frame if no validation data is provided? What happens if I use sample_rate=1.0 and do not provide a validation frame? $\endgroup$ – cryo111 Dec 19 '17 at 14:56
  • $\begingroup$ you should try the calculation again and set the max_depth =1 so you ensure only one split and variable importance for only one feature. H2O uses the improvement in squared error for each feature that was split on not the accuracy. Each features improvement is then summed up at the end to get its total feature importance (and then scaled between 0-1). $\endgroup$ – Lauren Jan 24 '18 at 21:37
  • $\begingroup$ @Lauren You say "H2O uses the improvement in squared error for each feature that was split on not the accuracy." - but how would the squared error for a categorical variable be calculated? Note that Species is my response variable. $\endgroup$ – cryo111 Jan 24 '18 at 22:28
  • $\begingroup$ @cryo111 these are great questions a bit hard to answer in a comment so let me point you to the two places in the documentation that explain how we do splitting for categoricals docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/gbm-faq/… and docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/gbm-faq/… how we split a categorical is different from splitting on a numeric feature but the calculation for squared error is the same. The docs are also being updated to help clarify this question. $\endgroup$ – Lauren Jan 25 '18 at 16:12
1
$\begingroup$

This isn't necessarily an answer, more so a question. I just can't comment yet :(

Could it be as simple as what the split decision is? In other words, if the tree says 1.75 as the split value, will that be the exact place it splits? Or is it rounding by chance, so that it's actually splitting at 1.7544 or something? Not sure if this would make the difference, but 0.29333 and 0.28585 are close enough together that it seems like the result of a rounding error... Just some thoughts. Sorry if it doesn't help.

$\endgroup$
  • 1
    $\begingroup$ Thanks for your help! The Petal.Width values are between 0.1 and 2.5 in 0.1 steps. I have been looking at the total POJO file - the Petal.Width split points are at 1.75 and 1.646875. The split points should, therefore, be far enough away from the Petal.Width levels so that the split decisions do not depend on rounding errors. $\endgroup$ – cryo111 Dec 12 '17 at 15:32
  • $\begingroup$ BTW: I tried to do the same with regression trees. There we know that h2o uses MSE reduction across nodes to calculate variable importance. Same story here, i.e., I get quite close but do not get a perfect match. It may indeed be a rounding issue when recording accuracy/SSR values or maybe some int by int division (like in python2). Who knows... My java skills are not good enough to understand the h2o source code. $\endgroup$ – cryo111 Dec 12 '17 at 15:42
  • $\begingroup$ Same scenario here: tried digging into the source code, but couldn't get to the variable importance equation :/ Possibly someone with better knowledge could find it! $\endgroup$ – creutzml Dec 12 '17 at 19:15
1
$\begingroup$

A little late but still can help some. I tested on almost the same sample as you, I just tested with only 1 selected variable (max_depth=1):

    library(h2o)
    data(iris)
    h2o.init()
    iris$Species=factor(ifelse(iris$Species=="virginica",
                             "virginica","other"))
    trainH2o<-as.h2o(iris)

    rf<-h2o.randomForest(x=colnames(iris)[1:4],
             y=colnames(iris)[5],
             training_frame = trainH2o,
             ntrees = 1,
             max_depth = 1,
             sample_rate=1.0)

    pojo=capture.output(h2o.download_pojo(rf))
    rule=pojo[grepl("double pred = ",pojo)]
    rule

This code returns:

    # [1] "    double pred =      (Double.isNaN(data[0]) || data[0 /* Sepal.Length */] <6.1492186f ? "

Let's build manually the "Random forest":

    var="Sepal.Length"
    val=6.1492186
    table(iris$Species)
table(iris[iris[,var]<val,]$Species)
    table(iris[iris[,var]>=val,]$Species)

We obtain a distribution of each node (other ; virginica): enter image description here

Then the relative importance from h2o is based on the following calculation: \begin{align} RelativeImportance&=\frac{100*50}{100+50}-(\frac{84*11}{84+11}+\frac{16*39}{16+39}) &=12.26156 \end{align}

Same result with H2O:

   h2o.varimp(rf)
   # Variable Importances: 
   #       variable relative_importance scaled_importance percentage
   # 1  Petal.Width           12.261563          1.000000   1.000000
   # 2 Sepal.Length            0.000000          0.000000   0.000000
   # 3  Sepal.Width            0.000000          0.000000   0.000000
   # 4 Petal.Length            0.000000          0.000000   0.000000

The relative importance of a variable as a general is certainly the sum of the previous function applied on each node where the variable is used.

I don't know how to interpret the function used, if someone can, it will be with pleasure.

Hope it helped!

$\endgroup$
  • $\begingroup$ Thx a lot! Will try to understand your equation... $\endgroup$ – cryo111 Nov 15 '18 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.