What are variable importance rankings useful for?

Question

I have become somewhat of a nihilist when it comes to variable importance rankings (in the context of multivariate models of all kinds).

Often in the course of my work, I am asked to either assist another team produce a variable importance ranking, or produce a variable importance ranking from my own work. In response to these requests, I ask the following questions

What would you like this variable importance ranking for? What do you hope to learn from it? What kind of decisions would you like to make using it?

The answers I receive almost always fall into one of two categories

• I would like to know the importance of the different variables in my model in predicting the response.
• I would like to use it for feature selection, by removing low importance variables.

The first response is tautological (I would like a variable importance ranking because I would like a variable importance ranking). I must assume that these rankings fill a psychological need when consuming the output of a multivariate model. I have a hard time understanding this, as ranking the variables "importance" individually seems to implicitly reject the multi-dimensional nature of the model in question.

The second response essentially reduces to an informal version of backwards selection, the statistical sins of which are well documented in other parts of CrossValidated.

I also struggle with the ill defined nature of importance rankings. There seems to be little agreement on what underlying concept the ranking should be measuring, giving them a very ad hoc flavor. There are many ways to assign an importance score or ranking, and they generally suffer from drawbacks and caveats:

• They can be highly algorithm dependent, as in the importance rankings in random forests and gbms.
• They can have extremely high variance, changing drastically with perturbations to the underlying data.
• They can suffer greatly from correlation in the input predictors.

So, with all that said, my question is, what are some statistically valid uses of variable importance rankings, or, what is a convincing argument (either to a statistician or a layman) for the futility of such a desire? I am interested in both general theoretical arguments and case studies, whichever would be more effective in making the point.

Answer 1

I have argued that variable importance is a slippery concept, as this question posits. The tautological first type of response that you get to your question and the unrealistic hopes of those who would interpret variable-importance results in terms of causality, as noted by @DexGroves, need little elaboration.

In fairness to those who would use backward selection, however, even Frank Harrell allows for it as part of a modeling strategy. From page 97 of his Regression Modeling Strategies, 2nd edition (a similar statement is on page 131 of the associated course notes):

13. Do limited backwards step-down variable selection if parsimony is more important than accuracy.

This limited potential use of backward selection, however, is step 13, the last step before the final model (step 14). It comes well after the crucial first steps:

1. Assemble as much accurate pertinent data as possible, with wide distributions for predictor values...
2. Formulate good hypotheses that lead to specification of relevant candidate predictors and possible interactions...

In my experience people often want to bypass step 2, and let some automated procedure replace intelligent application of subject-matter knowledge. This may lead to some of the emphasis placed on variable importance.

The full model of Harrell's step 14 is followed by 5 further steps of validation and adjustment, with a last step:

20. Develop simplifications to the full model by approximating it to any desired degrees of accuracy.

As other answers have noted, there are issues of actionability, cost, and simplicity that enter into the practical application of modeling results. For example, if I develop a new cancer biomarker that improves prognostication but costs $100,000 per test, it might be difficult to convince insurers or the government to pay for the test unless it is spectacularly useful. So it's not unreasonable for someone to want to focus on variables that are "most important," or to simplify an accurate model into one that is somewhat less accurate but is easier or less expensive to implement.

But this variable selection and model simplification should be for a specific purpose, and I think that is where the difficulty arises. The issue is similar to assessing classification schemes solely on the basis of percent of cases correctly classified. Just as different classification errors can have different costs, different model simplification schemes can have different costs that balance against their hoped-for benefits.

So I think that the issue to focus on as the analyst is the ability to estimate and illustrate these costs and benefits reliably with statistical modeling procedures, rather than worrying too much about an abstract concept of statistically validity per se. For example, pages 157-8 of Harrell's class notes linked above has an example of using the bootstrap to show the vagaries of ranking predictors in least squares; similar results can be found for variables sets selected by LASSO.

If that type of variability in variable selection doesn't get in the way of a particular practical application of the model that's OK. The job is to estimate how much and what type of trouble that simplification will lead to.

Answer 2

This is completely anecdotal, but I've found variable importance useful in identifying mistakes or weaknesses in GBMs.

Variable importance gives you a kind of huge cross-sectional overview of the model that would be hard to get otherwise. Variables higher in the list are seeing more activity (whether or not they are more 'important' is another question). Often a poorly behaving predictor (for instance something forward-looking, or a high-cardinality factor) will shoot to the top.

If there's a big disagreement between intuition variable importance and GBM variable importance, there's usually some valuable knowledge to be gained or a mistake to be found.

I would add a third answer to the "why are you asking me for this?" question, which is "because I want to understand what's causal to my response". Eep.

Answer 3

Variable importance rankings have a definite role in the applied business world whenever there is a need to prioritize the potentially large number of inputs to a process, any process. This information provides direction in terms of a focused strategy for attacking a problem, working down from most to least important, e.g., process cost reduction, given that the variables are leveragable and not fixed or structural factors immune to manipulation. At the end of the day, this should result in an A/B test of some kind.

To your point however, Matt, and as with any ordinal rankings, minor nuances or differences between variables can be ambiguous or obscured, vitiating their usefulness.

Answer 4

I'm completely agree with you in the theoretical point of view. But in the practical point of view, variable importance is very useful.

Let's take an example in which an insurance company wants to reduce the number of questions in a questionnaire quantifying the risk of their clients. The more complicated the questionnaire is, the less likely clients buy their products. For that reason, they want to reduce the less useful questions when maintaining the level of risk quantification. The solution is often to use variable importance to determine which questions be deleted from the questionnaire (and having "more or less" the same prediction about the risk profile of the prospect).