1
$\begingroup$

Can I interpret the importance scores obtained from Random forest model similar to the Betas from Linear Regression?

For example, if I have an equation

$$ y=\alpha +\beta_1X_1 + \beta_2X_2,$$

a 1 unit change in $X_1$ is associated with a $\beta_1$ unit change in $y$. Does a similar analogy hold for Importance score?

$\endgroup$
3
$\begingroup$

No, variable importance in random forests is completely dissimilar to regression betas.

There are actually different measures of variable importance. Most of them rely on assessing whether out-of-bag accuracy decreases if a predictor is randomly permuted. The idea is that if accuracy remains the same if you shuffle a predictor randomly, then that predictor can't be all that important. The different measures typically differ in how they assess accuracy (Gini or other impurity, MSE etc.).

If you use R and the randomForest package, then ?importance yields (under "Details"):

Here are the definitions of the variable importance measures. The first measure is computed from permuting OOB data: For each tree, the prediction error on the out-of-bag portion of the data is recorded (error rate for classification, MSE for regression). Then the same is done after permuting each predictor variable. The difference between the two are then averaged over all trees, and normalized by the standard deviation of the differences. If the standard deviation of the differences is equal to 0 for a variable, the division is not done (but the average is almost always equal to 0 in that case).

The second measure is the total decrease in node impurities from splitting on the variable, averaged over all trees. For classification, the node impurity is measured by the Gini index. For regression, it is measured by residual sum of squares.

(Plus, you shouldn't interpret regression betas as variable importance. If at all, look at standardized coefficients.)

$\endgroup$
  • 1
    $\begingroup$ In addition it's good to bootstrap the entire process (a new outer loop) to check the precision of the variable importance measure. I'll bet in many cases it is not stable. $\endgroup$ – Frank Harrell Feb 8 '18 at 13:00
  • $\begingroup$ Thanks @ Stephan Kolassa I have a simple question on how the permutation of a predictor is done. As per my understanding, I have a OOB sample of size 100 and a predictor p1 which is allowed to takes values (1,2,3,4,5). Then I have 100^5 possible permutations of p1? The process is repeated across all other predictors with the other held constant and then averaged? Thanks is advance. $\endgroup$ – sayantan das adhikari Feb 15 '18 at 13:52
  • $\begingroup$ Typically, not all possible permutations are run, since this would be far too many. Instead, you'd use random permutations. For instance, randomForest::randomForest() in R uses the parameter nPerm to control how many such random permutations are used. $\endgroup$ – S. Kolassa - Reinstate Monica Feb 15 '18 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.