24
$\begingroup$

I have become somewhat of a nihilist when it comes to variable importance rankings (in the context of multivariate models of all kinds).

Often in the course of my work, I am asked to either assist another team produce a variable importance ranking, or produce a variable importance ranking from my own work. In response to these requests, I ask the following questions

What would you like this variable importance ranking for? What do you hope to learn from it? What kind of decisions would you like to make using it?

The answers I receive almost always fall into one of two categories

  • I would like to know the importance of the different variables in my model in predicting the response.
  • I would like to use it for feature selection, by removing low importance variables.

The first response is tautological (I would like a variable importance ranking because I would like a variable importance ranking). I must assume that these rankings fill a psychological need when consuming the output of a multivariate model. I have a hard time understanding this, as ranking the variables "importance" individually seems to implicitly reject the multi-dimensional nature of the model in question.

The second response essentially reduces to an informal version of backwards selection, the statistical sins of which are well documented in other parts of CrossValidated.

I also struggle with the ill defined nature of importance rankings. There seems to be little agreement on what underlying concept the ranking should be measuring, giving them a very ad hoc flavor. There are many ways to assign an importance score or ranking, and they generally suffer from drawbacks and caveats:

  • They can be highly algorithm dependent, as in the importance rankings in random forests and gbms.
  • They can have extremely high variance, changing drastically with perturbations to the underlying data.
  • They can suffer greatly from correlation in the input predictors.

So, with all that said, my question is, what are some statistically valid uses of variable importance rankings, or, what is a convincing argument (either to a statistician or a layman) for the futility of such a desire? I am interested in both general theoretical arguments and case studies, whichever would be more effective in making the point.

$\endgroup$
  • 1
    $\begingroup$ Using variable importance (from some sensible procedure) to filter out weak predictors doesn't seem like a terrible idea. Can you clarify why you think this is bad? $\endgroup$ – dsaxton Mar 18 '16 at 0:05
  • 3
    $\begingroup$ I suppose that in general I think that many statistical processes are not dominated by "important" predictors, by are the accumulation of many small effects. For example, the power of ridge regression could be explained by it explicitly acknowledging this structure. Said another way, what is the reason we should believe, a priori, in the concept of a "weak predictor", and why should we filter them out? And why should we use such an informal procedure to do so when glmnet is available? $\endgroup$ – Matthew Drury Mar 18 '16 at 0:08
  • 2
    $\begingroup$ In any field in which we are not expert, we want to know what's important to worry about! Many business and management books seem to be about explaining at length that you identify the important problems and focus on them (yes indeed). I suspect that miscommunication here usually starts with non-statistical people supposing that there's a way to quantify importance and that it is statistical people's job to know how to do that and not worry them with how difficult it is. I don't know how to be less general, but some of the discussion here seems to miss key points in your question. $\endgroup$ – Nick Cox Mar 23 '16 at 13:05
8
$\begingroup$

I have argued that variable importance is a slippery concept, as this question posits. The tautological first type of response that you get to your question and the unrealistic hopes of those who would interpret variable-importance results in terms of causality, as noted by @DexGroves, need little elaboration.

In fairness to those who would use backward selection, however, even Frank Harrell allows for it as part of a modeling strategy. From page 97 of his Regression Modeling Strategies, 2nd edition (a similar statement is on page 131 of the associated course notes):

  1. Do limited backwards step-down variable selection if parsimony is more important than accuracy.

This limited potential use of backward selection, however, is step 13, the last step before the final model (step 14). It comes well after the crucial first steps:

  1. Assemble as much accurate pertinent data as possible, with wide distributions for predictor values...
  2. Formulate good hypotheses that lead to specification of relevant candidate predictors and possible interactions...

In my experience people often want to bypass step 2, and let some automated procedure replace intelligent application of subject-matter knowledge. This may lead to some of the emphasis placed on variable importance.

The full model of Harrell's step 14 is followed by 5 further steps of validation and adjustment, with a last step:

  1. Develop simplifications to the full model by approximating it to any desired degrees of accuracy.

As other answers have noted, there are issues of actionability, cost, and simplicity that enter into the practical application of modeling results. For example, if I develop a new cancer biomarker that improves prognostication but costs $100,000 per test, it might be difficult to convince insurers or the government to pay for the test unless it is spectacularly useful. So it's not unreasonable for someone to want to focus on variables that are "most important," or to simplify an accurate model into one that is somewhat less accurate but is easier or less expensive to implement.

But this variable selection and model simplification should be for a specific purpose, and I think that is where the difficulty arises. The issue is similar to assessing classification schemes solely on the basis of percent of cases correctly classified. Just as different classification errors can have different costs, different model simplification schemes can have different costs that balance against their hoped-for benefits.

So I think that the issue to focus on as the analyst is the ability to estimate and illustrate these costs and benefits reliably with statistical modeling procedures, rather than worrying too much about an abstract concept of statistically validity per se. For example, pages 157-8 of Harrell's class notes linked above has an example of using the bootstrap to show the vagaries of ranking predictors in least squares; similar results can be found for variables sets selected by LASSO.

If that type of variability in variable selection doesn't get in the way of a particular practical application of the model that's OK. The job is to estimate how much and what type of trouble that simplification will lead to.

$\endgroup$
  • 2
    $\begingroup$ This is a great answer @EdM and is quite consistent with the opinions I have developed on the matter. I especially like your two points that 1) unacceptable predictors (for moral, regulatory, or business reasons) should be screened out before modeling, 2) final model simplification should be for a speicific, apriori defined purpose. These are essentially the points I am usually trying to unwind with the questions to my business partners. $\endgroup$ – Matthew Drury Mar 25 '16 at 16:14
  • $\begingroup$ I also agree with your final point, that it is important to illustrate to partners the inherent variance in the final selection procedure. In the context of LASSO, I've settled on using the bootstrap to estimate, for each predictor, $Pr(\beta \ne 0)$, and the conditional variance of the estimate, given that it is non-zero. What do you think of this, are there more appropriate ways to summarize this variance? $\endgroup$ – Matthew Drury Mar 25 '16 at 16:16
  • $\begingroup$ With that said, I still wonder if there is some underlying concept that the importance rankings are trying to capture, of if they are all just ad hoc attacks at an unclear statistical problem. $\endgroup$ – Matthew Drury Mar 25 '16 at 16:17
  • 1
    $\begingroup$ @MatthewDrury , Frank Harrell provides one principled way to evaluate "variable importance," based on the fraction of log-likelihood explained by each variable. That's not what less-sophisticated individuals probably mean by the phrase. Like you, I have used the fraction of times that LASSO chooses each predictor, among multiple bootstrap samples, as the best way I can think of to illustrate the vagaries of variable selection. That's mostly pushed me away from LASSO and toward ridge regression for moderate-scale problems. $\endgroup$ – EdM Mar 25 '16 at 17:38
8
$\begingroup$

This is completely anecdotal, but I've found variable importance useful in identifying mistakes or weaknesses in GBMs.

Variable importance gives you a kind of huge cross-sectional overview of the model that would be hard to get otherwise. Variables higher in the list are seeing more activity (whether or not they are more 'important' is another question). Often a poorly behaving predictor (for instance something forward-looking, or a high-cardinality factor) will shoot to the top.

If there's a big disagreement between intuition variable importance and GBM variable importance, there's usually some valuable knowledge to be gained or a mistake to be found.

I would add a third answer to the "why are you asking me for this?" question, which is "because I want to understand what's causal to my response". Eep.

$\endgroup$
4
$\begingroup$

Variable importance rankings have a definite role in the applied business world whenever there is a need to prioritize the potentially large number of inputs to a process, any process. This information provides direction in terms of a focused strategy for attacking a problem, working down from most to least important, e.g., process cost reduction, given that the variables are leveragable and not fixed or structural factors immune to manipulation. At the end of the day, this should result in an A/B test of some kind.

To your point however, Matt, and as with any ordinal rankings, minor nuances or differences between variables can be ambiguous or obscured, vitiating their usefulness.

$\endgroup$
  • $\begingroup$ I completely agree with the usefulness of variable ranking in many business cases. But here the concern of 'different algorithms give different rankings' remains unaddressed. Do you have any suggestion to address that? Also see my question here stats.stackexchange.com/q/251248/71287 and the comments below that. $\endgroup$ – Aliweb Dec 13 '16 at 21:49
  • 3
    $\begingroup$ @aliweb The issue of difference doesn't have a single, fixed, unitary solution. This point is as subtle as the distinction between hierarchies and heterarchies where global rankings are revealed to be, in fact, wholly local and transient. The best reviews of the literature on relative variable importance probably belong to Ulrike Groemping whose papers are quite comprehensive wrt the various metrics that are out there. In addition, her R module and method -- RELAMPO -- is as rigorous an approach to estimating relative importance as exists. $\endgroup$ – DJohnson Dec 13 '16 at 22:17
3
$\begingroup$

I'm completely agree with you in the theoretical point of view. But in the practical point of view, variable importance is very useful.

Let's take an example in which an insurance company wants to reduce the number of questions in a questionnaire quantifying the risk of their clients. The more complicated the questionnaire is, the less likely clients buy their products. For that reason, they want to reduce the less useful questions when maintaining the level of risk quantification. The solution is often to use variable importance to determine which questions be deleted from the questionnaire (and having "more or less" the same prediction about the risk profile of the prospect).

$\endgroup$
  • $\begingroup$ I completely agree with the usefulness of variable ranking in many business cases. But here the concern of 'different algorithms give different rankings' remains unaddressed. Do you have any suggestion to address that? Also see my question here stats.stackexchange.com/q/251248/71287 and the comments below that. $\endgroup$ – Aliweb Dec 13 '16 at 21:50
  • $\begingroup$ @aliweb: I think Matthew already provided you an excelent answer to your question. $\endgroup$ – Metariat Dec 14 '16 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.