2
$\begingroup$

Random forests have their variable importance calculated using one of two methods, of which permutation-based importance is considered better. In R's randomForest package, this returns a measure called %IncMSE (or percentage increase in mean squared error) for regression cases. The calculation is explained clearly by @SorenHavelundWelling in this answer:

%IncMSE is the most robust and informative measure. It is the increase in mse of predictions(estimated with out-of-bag-CV) as a result of variable j being permuted(values randomly shuffled).

  1. grow regression forest. Compute OOB-mse, name this mse0.
  2. for 1 to j var: permute values of column j, then predict and compute OOB-mse(j)
  3. %IncMSE of j'th is (mse(j)-mse0)/mse0 * 100%

This matches my prior understanding as well. But the package documentation explains things this way (emphasis mine):

The first measure [%IncMSE] is computed from permuting OOB data: For each tree, the prediction error on the out-of-bag portion of the data is recorded (error rate for classification, MSE for regression). Then the same is done after permuting each predictor variable. The difference between the two are then averaged over all trees, and normalized by the standard deviation of the differences. If the standard deviation of the differences is equal to 0 for a variable, the division is not done (but the average is almost always equal to 0 in that case).

This normalization step does not really make sense to me, and is omitted by the answer quoted above as well. Why is this part of the calculation? And does the resulting value retain the meaning of a percentage change after this normalization? It wouldn't seem like it to me.

$\endgroup$
0
$\begingroup$

This seems analogous to $t-value$s reported for the coefficient estimates in linear regression as explained in this answer. It is the observed difference (in prediction errors) represented in units of standard error. The greater the magnitude of t-value, the greater the evidence against a null hypothesis (the implicit null hypothesis here being that the individual feature being tested has no predictive information).

While it seems like no $p-value$s, etc. are reported here (as in conventional hypothesis testing), there is still utility to this division in that it measures the "magnitude" of the feature importance relative to the variation in your observed differences in prediction error by permuting the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.