I'm having some difficulty understanding how to interpret variable importance output from the Random Forest package. Mean decrease in accuracy is usually described as "the decrease in model accuracy from permuting the values in each feature".

Is this a statement about the feature as a whole or about specific values within the feature? In either case, is the Mean Decrease in Accuracy the number or proportion of observations that are incorrectly classified by removing the feature (or values from the feature) in question from the model?

Say we have the following model:

require(randomForest)
data(iris)
set.seed(1)
dat <- iris
dat$Species <- factor(ifelse(dat$Species=='virginica','virginica','other'))
model.rf <- randomForest(Species~., dat, ntree=25,
importance=TRUE, nodesize=5)
model.rf
varImpPlot(model.rf)

Call:
 randomForest(formula = Species ~ ., data = dat, ntree = 25,
 proximity = TRUE, importance = TRUE, nodesize = 5)

Type of random forest: classification
Number of trees: 25
No. of variables tried at each split: 2

        OOB estimate of  error rate: 3.33%
Confusion matrix:
          other virginica class.error
other        97         3        0.03
virginica     2        48        0.04

enter image description here

In this model, the OOB rate is rather low (around 5%). Yet, the Mean Decrease in Accuracy for the predictor (Petal.Length) with the highest value in this measure is only around 8.

Does this mean that removing Petal.Length from the model would only result in an additional misclassification of 8 or so observations on average?

How could the Mean Decrease in Accuracy for Petal.Length be so low, given that it's the highest in this measure, and thus the other variables have even lower values on this measure?

up vote 22 down vote accepted

"Is this a statement about the feature as a whole or about specific values within the feature?"

  • "Global" variable importance is the mean decrease of accuracy over all out-of-bag cross validated predictions, when a given variable is permuted after training, but before prediction. "Global" is implicit. Local variable importance is the mean decrease of accuracy by each individual out-of-bag cross validated prediction. Global variable importance is the most popular, as it is a single number per variable, easier to understand, and more robust as it is averaged over all predictions.

"In either case, is the Mean Decrease in Accuracy the number or proportion of observations that are incorrectly classified by removing the feature (or values from the feature) in question from the model?"

  • 1 train forest
  • 2 measure out-of-bag CV accuracy -> OOB_acc_base
  • 3 permute variable i
  • 4 measure out-of-bag CV accuracy -> OOB_acc_perm_i
  • 5 VI_i = - (OOB_acc_perm_i - OOB_acc_base)

-"Does this mean that removing Petal.Length from the model would only result in an additional misclassification of 8 or so observations on average?"

  • Yep. Both Petal.length and Petal.width alone has almost perfect linear separation. Thus the variables share redundant information and permuting only one does not obstruct the model.

"How could the Mean Decrease in Accuracy for Petal.Length be so low, given that it's the highest in this measure, and thus the other variables have even lower values on this measure?"

  • When a robust/regularized model is trained on redundant variables, it is quite resistant to permutations in single variables.

Mainly use variable importance mainly to rank the usefulness of your variables. A clear interpretation of the absolute values of variable importance is hard to do well.

GINI: GINI importance measures the average gain of purity by splits of a given variable. If the variable is useful, it tends to split mixed labeled nodes into pure single class nodes. Splitting by a permuted variables tend neither to increase nor decrease node purities. Permuting a useful variable, tend to give relatively large decrease in mean gini-gain. GINI importance is closely related to the local decision function, that random forest uses to select the best available split. Therefore, it does not take much extra time to compute. On the other hand, mean gini-gain in local splits, is not necessarily what is most useful to measure, in contrary to change of overall model performance. Gini importance is overall inferior to (permutation based) variable importance as it is relatively more biased, more unstable and tend to answer a more indirect question.

  • For interpretation of variable importance beyond simple ranking, check out : "Bivariate variable selection for classification problem" -Vivian W. Ng and Leo Breiman digitalassets.lib.berkeley.edu/sdtr/ucb/text/692.pdf – Soren Havelund Welling Feb 22 '16 at 15:35
  • Thanks so much for your response! I've seen some places describe the mean decrease in accuracy as the increase in the OOB error rate (so a percentage). The formula you posted also seems to suggest an error rate: (OOB_acc_perm_i - OOB_acc_base). But you are sure Mean Decrease in Accuracy is referring to number of incorrectly classified observations? – FlacoT Feb 22 '16 at 15:37
  • 1
    Remember the minus in front, as variable importance is a decrease. I was not too specific with the units, these could be expressed in % or pure ratios/proportions, does not matter. But yes as accuracy=1-error_rate, VI_i = error_rate_perm_i - error_rate_base. For regression the unit of permutation variable importance is typically decrease of %explained variance and the unit of gini importance is mean decrease of mean_square_error-gain. " But you are sure Mean Decrease in Accuracy is referring to number of incorrectly classified observations?" -No, accuracy is a fraction, not a count. – Soren Havelund Welling Feb 22 '16 at 15:56

Here is the description of the mean decrease in accuracy (MDA) from the help manual of randomForest:

The first measure is computed from permuting OOB data: For each tree, the prediction error on the out-of-bag portion of the data is recorded (error rate for classification, MSE for regression). Then the same is done after permuting each predictor variable. The difference between the two are then averaged over all trees, and normalized by the standard deviation of the differences. If the standard deviation of the differences is equal to 0 for a variable, the division is not done (but the average is almost always equal to 0 in that case).

According to the description, the "accuracy" in MDA actually refers to the accuracy of single tree models, regardless of the fact that we are more concerned with the error rate of the forest. So,

"Does this mean that removing Petal.Length from the model would only result in an additional misclassification of 8 or so observations on average?"

  • First, the MDA (scaled by default) as defined above is more like a test statistic: $$ \frac{\text{Mean(Decreases in Accuracy of Trees)}} {\text{StandardDeviation(Decreases in Accuracy of Trees)}} $$ The scale is neither percentage or count of observations.

  • Second, even the unscaled MDA, i.e. $\text{Mean(Decreases in Accuracy of Trees)}$, doesn't tell anything about the accuracy of the forest model (trees as a whole by voting).

In summary, the MDA output by randomForest package is neither about error rate nor error counts, but better interpreted as a test statistic on the hypothesis test: $$ H_0: \text{Nodes constructed by predictor } i \text{ is useless in any single trees} $$ versus $$ H_1: \text{Nodes constructed by predictor } i \text{ is useful} $$

As a remark, the MDA procedure described by Soren is different from the implementation of randomForest package. It is closer to what we desire from an MDA: the accuracy decrease of the whole forest model. However, the model will probably be fitted differently without Petal.Length and rely more on other predictors. Thus Soren's MDA would be too pessimistic.

  • Two follow-up questions: 1. Any idea if other packages use the more intuitive MDA described by @Soren? 2. If the interpretation of MDA in RandomForest is as a test statistic, is there anything like a rule-of-thumb on what a sufficiently large test statistic is to reject H0? Does MDA follow some known distribution? – FlacoT Dec 6 '16 at 19:01
  • 1. Sorry, I didn't try any other package. 2. It is simply alike a test statistic. Neither the distribution is accessible (as far as I know few people looked into this) nor the test itself meaningful--I don't think the test concludes anything about the FOREST, which is our actual interest. – Jianyu Dec 7 '16 at 20:05

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.