# How to interpret Mean Decrease in Accuracy and Mean Decrease GINI in Random Forest models

I'm having some difficulty understanding how to interpret variable importance output from the Random Forest package. Mean decrease in accuracy is usually described as "the decrease in model accuracy from permuting the values in each feature".

Is this a statement about the feature as a whole or about specific values within the feature? In either case, is the Mean Decrease in Accuracy the number or proportion of observations that are incorrectly classified by removing the feature (or values from the feature) in question from the model?

Say we have the following model:

require(randomForest)
data(iris)
set.seed(1)
dat <- iris
dat$Species <- factor(ifelse(dat$Species=='virginica','virginica','other'))
model.rf <- randomForest(Species~., dat, ntree=25,
importance=TRUE, nodesize=5)
model.rf
varImpPlot(model.rf)


Call:
randomForest(formula = Species ~ ., data = dat, ntree = 25,
proximity = TRUE, importance = TRUE, nodesize = 5)

Type of random forest: classification
Number of trees: 25
No. of variables tried at each split: 2

OOB estimate of  error rate: 3.33%
Confusion matrix:
other virginica class.error
other        97         3        0.03
virginica     2        48        0.04


In this model, the OOB rate is rather low (around 5%). Yet, the Mean Decrease in Accuracy for the predictor (Petal.Length) with the highest value in this measure is only around 8.

Does this mean that removing Petal.Length from the model would only result in an additional misclassification of 8 or so observations on average?

How could the Mean Decrease in Accuracy for Petal.Length be so low, given that it's the highest in this measure, and thus the other variables have even lower values on this measure?

• try dat <- iris[,c(2:3,5)] and should get higher VI-values – Soren Havelund Welling Feb 22 '16 at 15:04

"Is this a statement about the feature as a whole or about specific values within the feature?"

• "Global" variable importance is the mean decrease of accuracy over all out-of-bag cross validated predictions, when a given variable is permuted after training, but before prediction. "Global" is implicit. Local variable importance is the mean decrease of accuracy by each individual out-of-bag cross validated prediction. Global variable importance is the most popular, as it is a single number per variable, easier to understand, and more robust as it is averaged over all predictions.

"In either case, is the Mean Decrease in Accuracy the number or proportion of observations that are incorrectly classified by removing the feature (or values from the feature) in question from the model?"

1. train forest
2. measure out-of-bag CV accuracy → OOB_acc_base
3. permute variable i
4. measure out-of-bag CV accuracy → OOB_acc_perm_i
5. VI_i = - (OOB_acc_perm_i - OOB_acc_base)

-"Does this mean that removing Petal.Length from the model would only result in an additional misclassification of 8 or so observations on average?"

• Yep. Both Petal.length and Petal.width alone has almost perfect linear separation. Thus the variables share redundant information and permuting only one does not obstruct the model.

"How could the Mean Decrease in Accuracy for Petal.Length be so low, given that it's the highest in this measure, and thus the other variables have even lower values on this measure?"

• When a robust/regularized model is trained on redundant variables, it is quite resistant to permutations in single variables.

Mainly use variable importance mainly to rank the usefulness of your variables. A clear interpretation of the absolute values of variable importance is hard to do well.

GINI: GINI importance measures the average gain of purity by splits of a given variable. If the variable is useful, it tends to split mixed labeled nodes into pure single class nodes. Splitting by a permuted variables tend neither to increase nor decrease node purities. Permuting a useful variable, tend to give relatively large decrease in mean gini-gain. GINI importance is closely related to the local decision function, that random forest uses to select the best available split. Therefore, it does not take much extra time to compute. On the other hand, mean gini-gain in local splits, is not necessarily what is most useful to measure, in contrary to change of overall model performance. Gini importance is overall inferior to (permutation based) variable importance as it is relatively more biased, more unstable and tend to answer a more indirect question.

• For interpretation of variable importance beyond simple ranking, check out : "Bivariate variable selection for classification problem" -Vivian W. Ng and Leo Breiman digitalassets.lib.berkeley.edu/sdtr/ucb/text/692.pdf – Soren Havelund Welling Feb 22 '16 at 15:35
• Thanks so much for your response! I've seen some places describe the mean decrease in accuracy as the increase in the OOB error rate (so a percentage). The formula you posted also seems to suggest an error rate: (OOB_acc_perm_i - OOB_acc_base). But you are sure Mean Decrease in Accuracy is referring to number of incorrectly classified observations? – FlacoT Feb 22 '16 at 15:37
• Remember the minus in front, as variable importance is a decrease. I was not too specific with the units, these could be expressed in % or pure ratios/proportions, does not matter. But yes as accuracy=1-error_rate, VI_i = error_rate_perm_i - error_rate_base. For regression the unit of permutation variable importance is typically decrease of %explained variance and the unit of gini importance is mean decrease of mean_square_error-gain. " But you are sure Mean Decrease in Accuracy is referring to number of incorrectly classified observations?" -No, accuracy is a fraction, not a count. – Soren Havelund Welling Feb 22 '16 at 15:56

Here is the description of the mean decrease in accuracy (MDA) from the help manual of randomForest:

The first measure is computed from permuting OOB data: For each tree, the prediction error on the out-of-bag portion of the data is recorded (error rate for classification, MSE for regression). Then the same is done after permuting each predictor variable. The difference between the two are then averaged over all trees, and normalized by the standard deviation of the differences. If the standard deviation of the differences is equal to 0 for a variable, the division is not done (but the average is almost always equal to 0 in that case).

According to the description, the "accuracy" in MDA actually refers to the accuracy of single tree models, regardless of the fact that we are more concerned with the error rate of the forest. So,

"Does this mean that removing Petal.Length from the model would only result in an additional misclassification of 8 or so observations on average?"

• First, the MDA (scaled by default) as defined above is more like a test statistic: $$\frac{\text{Mean(Decreases in Accuracy of Trees)}} {\text{StandardDeviation(Decreases in Accuracy of Trees)}}$$ The scale is neither percentage or count of observations.

• Second, even the unscaled MDA, i.e. $\text{Mean(Decreases in Accuracy of Trees)}$, doesn't tell anything about the accuracy of the forest model (trees as a whole by voting).

In summary, the MDA output by randomForest package is neither about error rate nor error counts, but better interpreted as a test statistic on the hypothesis test: $$H_0: \text{Nodes constructed by predictor } i \text{ is useless in any single trees}$$ versus $$H_1: \text{Nodes constructed by predictor } i \text{ is useful}$$

As a remark, the MDA procedure described by Soren is different from the implementation of randomForest package. It is closer to what we desire from an MDA: the accuracy decrease of the whole forest model. However, the model will probably be fitted differently without Petal.Length and rely more on other predictors. Thus Soren's MDA would be too pessimistic.

• Two follow-up questions: 1. Any idea if other packages use the more intuitive MDA described by @Soren? 2. If the interpretation of MDA in RandomForest is as a test statistic, is there anything like a rule-of-thumb on what a sufficiently large test statistic is to reject H0? Does MDA follow some known distribution? – FlacoT Dec 6 '16 at 19:01
• 1. Sorry, I didn't try any other package. 2. It is simply alike a test statistic. Neither the distribution is accessible (as far as I know few people looked into this) nor the test itself meaningful--I don't think the test concludes anything about the FOREST, which is our actual interest. – Jianyu Dec 7 '16 at 20:05

A recent (blog post)[https://explained.ai/rf-importance/index.html] from a team at the University of San Francisco shows that default importance strategies in both R (randomForest) and Python (scikit) are unreliable in many data scenarios. Particularly, mean decrease in impurity importance metrics are biased when potential predictor variables vary in their scale of measurement or their number of categories.

The papers and blog post demonstrate how continuous and high cardinality variables are preferred in mean decrease in impurity importance rankings, even if they are equally uninformative compared to variables with less categories. The authors suggest using permutation importance instead of the default in these cases. If the predictor variables in your model are highly correlated, conditional permutation importance is suggested.

The impurity is biased since at each time a breakpoint is selected in a variable, every level of the variable is tested to find the best break point. Continuous or high cardinality variables will have many more split points, which results in the “multiple testing” problem. That is, there is a higher probability that by chance that variable happens to predict the outcome well, since variables, where more splits are tried, will appear more often in the tree.