At the model level, to assess predictor contribution/importance we can use:
- Model Specific Techniques – e.g. purity (Gini Index) for a tree-based model, model coefficients where applicable etc.
- Model Independent Techniques – e.g. Permutation Feature Importance, Partial Dependence etc.
What this does not convey is for a particular prediction (say a binary classification that provides a 92% probability of membership of class 1) what predictors were most “influential” in producing that prediction.
Having thought about this problem a little, it seems to me there are a few approaches that could be taken:
- Model Specific Techniques – e.g. coefficients of applicable linear models, techniques such as described here for say XGBoost (https://medium.com/applied-data-science/new-r-package-the-xgboost-explainer-51dd7d1aa211)
- Model Independent Techniques – e.g. some kind of “perturbation method” similar to Partial Dependence to understand how the prediction changes when we perturb the predictor and perhaps model that?, or techniques like LIME described in this paper (https://arxiv.org/pdf/1602.04938.pdf and https://github.com/marcotcr/lime), a modified Permutation Importance technique?
It seems to me the most valuable approach would be a model independent technique given the somewhat “black-box” nature of many algorithms, and providing an ability to interpret novel and new algorithms and techniques.
One naive method, described here (http://amunategui.github.io/actionable-instights/index.html) is to take each predictor, “neutralise” its impact by say imputing the "population" mean, and run the prediction again getting a difference between the original prediction and the neutralised version providing an importance measure. This seems a special case of a kind “perturbation” method hinted at above. A couple of flaws I see in this are that 1) it seems to imply that a prediction that has the “mean” (or equivalent) of each feature necessarily is a “middle” prediction perhaps, and 2) that features that are “means” (or equivalent) are necessarily non-impactful?
More generally any technique would have to account for:
- How to handle different data types (numerical, categorical etc.)
- How to handle missing data
- How to handle conditional importance perhaps (i.e. that predictors may only be important in pairs etc.)
- Computational efficiency (is it really practical to run a prediction $p$ times where $p$ is the number of predictors, or for a perturbation method $kp$ where $k$ is the number of predictions per predictor etc.)
With those loose and perhaps incorrect thoughts on the matter laid down, I wonder what approaches to the problem people are aware of, have considered, have used, would advise etc.?