Searching this site, I see over 1,000 posts triggered by the search term "variable importance", mostly machine learning related. However, I've never encountered the definition before. Is it a term that applies to a specific (set of) model(s)? If so, what does it measure and how is it measured? Furthermore, does it mean "importance" in any broader set of criteria than a statistical result?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
I've only encountered the term in machine learning contexts (that is, contexts where one is interested in accurate predictions and not necessarily theoretical inferences), but the concept can be applied to any statistical model.
(My) definition: Variable importance refers to how much a given model "uses" that variable to make accurate predictions. The more a model relies on a variable to make predictions, the more important it is for the model.
It can apply to many different models, each using different metrics.
Imagine two variables on the same scale in a standard ordinary least squares regression. One has a regression coefficient of 1.6, the other has one of .003. The former is a more important variable than the latter, because the model relies in the former more (remember that the variables are on the same scale and their coefficients are directly comparable). Another way to do this would be to look at the change in $R^2$ when adding each variable; the one with a higher $\Delta R^2$ is more important.
Similarly, one could compare two variables used in a random forest. If the trees in the forest split the sample more on variable A than variable B, then variable A is more important to the model. There are a bunch of metrics for quantifying this, for example, here is what the documentation for the popular randomForest::importance() package does:
Here are the definitions of the variable importance measures. The first measure is computed from permuting OOB data: For each tree, the prediction error on the out-of-bag portion of the data is recorded (error rate for classification, MSE for regression). Then the same is done after permuting each predictor variable. The difference between the two are then averaged over all trees, and normalized by the standard deviation of the differences. If the standard deviation of the differences is equal to 0 for a variable, the division is not done (but the average is almost always equal to 0 in that case). The second measure is the total decrease in node impurities from splitting on the variable, averaged over all trees. For classification, the node impurity is measured by the Gini index. For regression, it is measured by residual sum of squares.
Variable importance is often used for variable selection: What variables could we drop from the model (not contributing much information), and what variables should we make sure to always measure and use in the model?
The wonderful Introduction to Statistical Learning book by James, Witten, Hastie, & Tibshirani specifically discusses variable importance a few times throughout the book (e.g., p. 319, 330). URL: http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf
answered Mar 12, 2018 at 15:14
-
$\begingroup$ Hmm, but the "importance" of a variable depends on the variables that are included in the model, or in some cases the order in which they're entered into the model. I seem to see references to "importance" when considering many variables, not just a single one. Must it accompany some concept of a priority ranking in which (an) other variable(s) is/are entered into a model? $\endgroup$– AdamOMar 12, 2018 at 15:46
-
$\begingroup$ Yes, importance can depend on other variables entered into the model. So multicollinearity might suppress the importance of variables in some models. I think it's up to the analyst to use a correct metric for the type of model they are using—and there are a wiiide variety of importance metrics floating out there (scholar.google.com/…). And yes, since people almost always using more than one variable in a model in machine learning, it is a relative measure at heart. $\endgroup$ Mar 12, 2018 at 16:21
-
$\begingroup$ @MarkWhite +1 , what does the y axis represent? Is it a percentage? Or could I get a variable which had an importance of 9 from the output of random forest. edit Also - what about the result of negative importance within the context of binary classification? thanks $\endgroup$– baxxApr 18, 2019 at 21:47
-
1$\begingroup$ Does the order in which variables are added into the model relevant when assessing the increase of R-squared? $\endgroup$ Nov 30, 2021 at 18:01