1
$\begingroup$

I implemented a random forest model in R using the package 'ranger' combined in 'caret' package with 10fold CV. My outcome is binary (0,1) and I have a couple numeric predictor variables. I used the Gini index to evaluate variable importance, but I only got the overall results.

How can I separate the overall variable importance values for the two outcomes (0 and 1) to correctly interpret the relationship between each numeric predictor and the binary outcome? (e.g. to decide whether the increase or the decrease of one important numeric predictor has an impact on the outcome '1' )?

Do I need to set a specific argument or calculate other indices?

$\endgroup$

1 Answer 1

2
$\begingroup$

Gini importance and other impurity related measures usually used in Random Forests to estimate variable importance (aka feature importance) cannot provide that. The reason is the way it is defined:

For the impurity importance, a split with a large decrease of impurity is considered important and as a consequence variables used for splitting at important splits are also considered important. Based on this idea, the impurity importance for a variable $X_i$ is computed by the sum of all impurity decrease measures of all nodes in the forest at which a split on $X_i$ has been conducted, normalized by the number of trees.

(see "The revival of the Gini importance?"; Nembrini, König, Wright; Bioinformatics, 34(21), 2018, 3711–3718)

As you can see from this definition, there is no relation to what the prediction will be or to the increase or decrease of a predictor. Now, if we look one level deeper, i.e. how the impurity decrease is calculated for each node, we can see that it will not provide the desired information either.

Let $\hat{\phi}_j(t)$ be the class frequency for class $j$ in node $t$ then its impurity is

$$\hat{\Gamma}(t) = \sum_{j=1}^{J} \hat{\phi}_j(t)(1- \hat{\phi}_j(t)).$$ Now let $t_L$ be the left and let $t_R$ be the right child of a given node $t$. Then the impurity decrease of a node $t$ is calculated as the difference between its own impurity and the impurity of its left and right child weighted by the number of data points in each child node: $$\hat{\Gamma}(t) - (\hat{p}(t_L))\hat{\Gamma}(t_L) + \hat{p}(t_R)\hat{\Gamma}(t_R))$$ Where $\hat{p}(t_L)$ and $\hat{p}(t_R)$ refer to the left and right share of the data points split at node $t$. (see "The effect of splitting on random forests"; Ishwaran; Mach Learn (2015) 99:75–118) So basically this is just the difference between the impurity of the original node and its childs.

And as you can see also on the node level the calculation is not related to specific values of the prediction or whether a predictor is increased or decreased. It just estimates the contribution of a node (and thereby the corresponding variable or variable split point for non-binary variables) to the purity of the tree. Which is in the end aggregated across the forest to provide the overall variable importance.

On a side note: Besides the above mentioned limitations Gini importance specifically has been discussed as being biased towards categorical variables with larger domains as well as continuous variables. (see "Bias in random forest variable importance measures: Illustrations, sources and a solution"; Strobl, Boulesteix, Zeileis, Hothorn; BMC Bioinformatics 2007, 8:25)

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. But is there any efficient way to evaluate these relationships from random forest calculations? I recently read chapters from the book "Applied Predictive Modeling; Kuhn, Johnson (2013), and found that using AUC (using predictor data as input to ROC) can be appropriate to quantify predictor relevance, but I still did not gain any clear insights how to evaluate the effect of each predictor on the outcome from the random forest results (or it is possible at all). $\endgroup$
    – Adam F
    Dec 3, 2019 at 17:20
  • $\begingroup$ Unfortunately I am not aware of any approach for that. Random Forests already provide relatively high transparency through the different ways to measure their features importances (compared to other machine learning techniques like artificial neural networks). But I haven't seen anything going beyond this "global one-dimensional" feature importance you get. $\endgroup$
    – Jonathan
    Dec 3, 2019 at 22:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.