I have become somewhat of a nihilist when it comes to variable importance rankings (in the context of multivariate models of all kinds).
Often in the course of my work, I am asked to either assist another team produce a variable importance ranking, or produce a variable importance ranking from my own work. In response to these requests, I ask the following questions
What would you like this variable importance ranking for? What do you hope to learn from it? What kind of decisions would you like to make using it?
The answers I receive almost always fall into one of two categories
- I would like to know the importance of the different variables in my model in predicting the response.
- I would like to use it for feature selection, by removing low importance variables.
The first response is tautological (I would like a variable importance ranking because I would like a variable importance ranking). I must assume that these rankings fill a psychological need when consuming the output of a multivariate model. I have a hard time understanding this, as ranking the variables "importance" individually seems to implicitly reject the multi-dimensional nature of the model in question.
The second response essentially reduces to an informal version of backwards selection, the statistical sins of which are well documented in other parts of CrossValidated.
I also struggle with the ill defined nature of importance rankings. There seems to be little agreement on what underlying concept the ranking should be measuring, giving them a very ad hoc flavor. There are many ways to assign an importance score or ranking, and they generally suffer from drawbacks and caveats:
- They can be highly algorithm dependent, as in the importance rankings in random forests and gbms.
- They can have extremely high variance, changing drastically with perturbations to the underlying data.
- They can suffer greatly from correlation in the input predictors.
So, with all that said, my question is, what are some statistically valid uses of variable importance rankings, or, what is a convincing argument (either to a statistician or a layman) for the futility of such a desire? I am interested in both general theoretical arguments and case studies, whichever would be more effective in making the point.