How to interpret Mean Decrease in Accuracy and Mean Decrease GINI in Random Forest models I'm having some difficulty understanding how to interpret variable importance output from the Random Forest package. Mean decrease in accuracy is usually described as "the decrease in model accuracy from permuting the values in each feature".
Is this a statement about the feature as a whole or about specific values within the feature? In either case, is the Mean Decrease in Accuracy the number or proportion of observations that are incorrectly classified by removing the feature (or values from the feature) in question from the model?
Say we have the following model:
require(randomForest)
data(iris)
set.seed(1)
dat <- iris
dat$Species <- factor(ifelse(dat$Species=='virginica','virginica','other'))
model.rf <- randomForest(Species~., dat, ntree=25,
importance=TRUE, nodesize=5)
model.rf
varImpPlot(model.rf)


Call:
 randomForest(formula = Species ~ ., data = dat, ntree = 25,
 proximity = TRUE, importance = TRUE, nodesize = 5)

Type of random forest: classification
Number of trees: 25
No. of variables tried at each split: 2

        OOB estimate of  error rate: 3.33%
Confusion matrix:
          other virginica class.error
other        97         3        0.03
virginica     2        48        0.04


In this model, the OOB rate is rather low (around 5%). Yet, the Mean Decrease in Accuracy for the predictor (Petal.Width) with the highest value in this measure is only around 8.
Does this mean that removing Petal.Width from the model would only result in an additional misclassification of approximately 8 observations on average?
How could the Mean Decrease in Accuracy for Petal.Length be so low, given that it's the highest in this measure, and thus the other variables have even lower values on this measure?
 A: "Is this a statement about the feature as a whole or about specific values within the feature?"


*

*"Global" variable importance is the mean decrease of accuracy over all out-of-bag cross validated predictions, when a given variable is permuted after training, but before prediction. "Global" is implicit. Local variable importance is the mean decrease of accuracy by each individual out-of-bag cross validated prediction. Global variable importance is the most popular, as it is a single number per variable, easier to understand, and more robust as it is averaged over all predictions.


"In either case, is the Mean Decrease in Accuracy the number or proportion of observations that are incorrectly classified by removing the feature (or values from the feature) in question from the model?"


*

*train forest

*measure out-of-bag CV accuracy → OOB_acc_base

*permute variable i

*measure out-of-bag CV accuracy → OOB_acc_perm_i

*VI_i = - (OOB_acc_perm_i - OOB_acc_base)


-"Does this mean that removing Petal.Length from the model would only result in an additional misclassification of 8 or so observations on average?"


*

*Yep. Both Petal.length and Petal.width alone has almost perfect linear separation. Thus the variables share redundant information and permuting only one does not obstruct the model.


"How could the Mean Decrease in Accuracy for Petal.Length be so low, given that it's the highest in this measure, and thus the other variables have even lower values on this measure?"


*

*When a robust/regularized model is trained on redundant variables, it is quite resistant to permutations in single variables.


Mainly use variable importance mainly to rank the usefulness of your variables. A clear interpretation of the absolute values of variable importance is hard to do well.
GINI:
GINI importance measures the average gain of purity by splits of a given variable. If the variable is useful, it tends to split mixed labeled nodes into pure single class nodes. Splitting by a permuted variables tend neither to increase nor decrease node purities. Permuting a useful variable, tend to give relatively large decrease in mean gini-gain. GINI importance is closely related to the local decision function, that random forest uses to select the best available split. Therefore, it does not take much extra time to compute. On the other hand, mean gini-gain in local splits, is not necessarily what is most useful to measure, in contrary to change of overall model performance. Gini importance is overall inferior to (permutation based) variable importance as it is relatively more biased, more unstable and tend to answer a more indirect question.
A: Here is the description of the mean decrease in accuracy (MDA) from the help manual of randomForest:

The first measure is computed from permuting OOB data: For each tree, the prediction error on the out-of-bag portion of the data is recorded (error rate for classification, MSE for regression). Then the same is done after permuting each predictor variable. The difference between the two are then averaged over all trees, and normalized by the standard deviation of the differences. If the standard deviation of the differences is equal to 0 for a variable, the division is not done (but the average is almost always equal to 0 in that case). 

According to the description, the "accuracy" in MDA actually refers to the accuracy of single tree models, regardless of the fact that we are more concerned with the error rate of the forest. So,

"Does this mean that removing Petal.Length from the model would only result in an additional misclassification of 8 or so observations on average?" 



*

*First, the MDA (scaled by default) as defined above is more like a test statistic:
$$
\frac{\text{Mean(Decreases in Accuracy of Trees)}}  {\text{StandardDeviation(Decreases in Accuracy of Trees)}}
$$
The scale is neither percentage or count of observations.

*Second, even the unscaled MDA, i.e. $\text{Mean(Decreases in Accuracy of Trees)}$, doesn't tell anything about the accuracy of the forest model (trees as a whole by voting).
In summary, the MDA output by randomForest package is neither about error rate nor error counts, but better interpreted as a test statistic on the hypothesis test: 
$$
H_0: \text{Nodes constructed by predictor } i \text{ is useless in any single trees}
$$
versus 
$$
H_1: \text{Nodes constructed by predictor } i \text{ is useful}
$$
As a remark, the MDA procedure described by Soren is different from the implementation of randomForest package. It is closer to what we desire from an MDA: the accuracy decrease of the whole forest model. However, the model will probably be fitted differently without Petal.Length and rely more on other predictors. Thus Soren's MDA would be too pessimistic.
A: A recent blog post from a team at the University of San Francisco shows that default importance strategies in both R (randomForest) and Python (scikit) are unreliable in many data scenarios. Particularly, mean decrease in impurity importance metrics are biased when potential predictor variables vary in their scale of measurement or their number of categories.
The papers and blog post demonstrate how continuous and high cardinality variables are preferred in mean decrease in impurity importance rankings, even if they are equally uninformative compared to variables with less categories. The authors suggest using permutation importance instead of the default in these cases. If the predictor variables in your model are highly correlated, conditional permutation importance is suggested.
The impurity is biased since at each time a breakpoint is selected in a variable, every level of the variable is tested to find the best break point. Continuous or high cardinality variables will have many more split points, which results in the “multiple testing” problem. That is, there is a higher probability that by chance that variable happens to predict the outcome well, since variables, where more splits are tried, will appear more often in the tree.
A: Relative Mean Decrease Accuracy?

"In either case, is the Mean Decrease in Accuracy the number or proportion of observations that are incorrectly classified by removing the feature (or values from the feature) in question from the model?"



*

*train forest

*measure out-of-bag CV accuracy → OOB_acc_base

*permute variable i

*measure out-of-bag CV accuracy → OOB_acc_perm_i

*VI_i = - (OOB_acc_perm_i - OOB_acc_base)


So, Ugo Mela if this shows an absolute value of incorrectly classified observations, if we divide the Mean Decrease in Accuracy (MDA) by the total number of observations with that feature then we can get a mean relative importance of each feature?

*

*1000 observations of feature A;

*MDA(feature A) = 500;

*500/1000 * 100 = 50%.

With this we can look which features, even if in low amounts, greatly improve the model.
