Searching this site, I see over 1,000 posts triggered by the search term "variable importance", mostly machine learning related. However, I've never encountered the definition before. Is it a term that applies to a specific (set of) model(s)? If so, what does it measure and how is it measured? Furthermore, does it mean "importance" in any broader set of criteria than a statistical result?
I've only encountered the term in machine learning contexts (that is, contexts where one is interested in accurate predictions and not necessarily theoretical inferences), but the concept can be applied to any statistical model.
(My) definition: Variable importance refers to how much a given model "uses" that variable to make accurate predictions. The more a model relies on a variable to make predictions, the more important it is for the model.
It can apply to many different models, each using different metrics.
Imagine two variables on the same scale in a standard ordinary least squares regression. One has a regression coefficient of 1.6, the other has one of .003. The former is a more important variable than the latter, because the model relies in the former more (remember that the variables are on the same scale and their coefficients are directly comparable). Another way to do this would be to look at the change in $R^2$ when adding each variable; the one with a higher $\Delta R^2$ is more important.
Similarly, one could compare two variables used in a random forest. If the trees in the forest split the sample more on variable A than variable B, then variable A is more important to the model. There are a bunch of metrics for quantifying this, for example, here is what the documentation for the popular
randomForest::importance() package does:
Here are the definitions of the variable importance measures. The first measure is computed from permuting OOB data: For each tree, the prediction error on the out-of-bag portion of the data is recorded (error rate for classification, MSE for regression). Then the same is done after permuting each predictor variable. The difference between the two are then averaged over all trees, and normalized by the standard deviation of the differences. If the standard deviation of the differences is equal to 0 for a variable, the division is not done (but the average is almost always equal to 0 in that case). The second measure is the total decrease in node impurities from splitting on the variable, averaged over all trees. For classification, the node impurity is measured by the Gini index. For regression, it is measured by residual sum of squares.
Variable importance is often used for variable selection: What variables could we drop from the model (not contributing much information), and what variables should we make sure to always measure and use in the model?
The wonderful Introduction to Statistical Learning book by James, Witten, Hastie, & Tibshirani specifically discusses variable importance a few times throughout the book (e.g., p. 319, 330). URL: http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf