Feature Importance/Impact for Individual Predictions

Question

At the model level, to assess predictor contribution/importance we can use:

• Model Specific Techniques – e.g. purity (Gini Index) for a tree-based model, model coefficients where applicable etc.
• Model Independent Techniques – e.g. Permutation Feature Importance, Partial Dependence etc.

What this does not convey is for a particular prediction (say a binary classification that provides a 92% probability of membership of class 1) what predictors were most “influential” in producing that prediction.

Having thought about this problem a little, it seems to me there are a few approaches that could be taken:

• Model Specific Techniques – e.g. coefficients of applicable linear models, techniques such as described here for say XGBoost (https://medium.com/applied-data-science/new-r-package-the-xgboost-explainer-51dd7d1aa211)
• Model Independent Techniques – e.g. some kind of “perturbation method” similar to Partial Dependence to understand how the prediction changes when we perturb the predictor and perhaps model that?, or techniques like LIME described in this paper (https://arxiv.org/pdf/1602.04938.pdf and https://github.com/marcotcr/lime), a modified Permutation Importance technique?

It seems to me the most valuable approach would be a model independent technique given the somewhat “black-box” nature of many algorithms, and providing an ability to interpret novel and new algorithms and techniques.

One naive method, described here (http://amunategui.github.io/actionable-instights/index.html) is to take each predictor, “neutralise” its impact by say imputing the "population" mean, and run the prediction again getting a difference between the original prediction and the neutralised version providing an importance measure. This seems a special case of a kind “perturbation” method hinted at above. A couple of flaws I see in this are that 1) it seems to imply that a prediction that has the “mean” (or equivalent) of each feature necessarily is a “middle” prediction perhaps, and 2) that features that are “means” (or equivalent) are necessarily non-impactful?

More generally any technique would have to account for:

• How to handle different data types (numerical, categorical etc.)
• How to handle missing data
• How to handle conditional importance perhaps (i.e. that predictors may only be important in pairs etc.)
• Computational efficiency (is it really practical to run a prediction $p$ times where $p$ is the number of predictors, or for a perturbation method $kp$ where $k$ is the number of predictions per predictor etc.)

With those loose and perhaps incorrect thoughts on the matter laid down, I wonder what approaches to the problem people are aware of, have considered, have used, would advise etc.?

Answer 1

The topic you are addressing is known as model explanation or model interpretation and quite an active topic in research. The general idea is to find out, which features contributed to the model, and which not.

You already mentioned some popular techniques, such as the Partial Dependence Plots (PDP) or LIME. In a PDP, the influence of a features' value to the model output is displayed by creating new instances from the data that have a modified feature value and predict them by the model. LIME creates a local approximation of the model by sampling instances around a requested instance and learning a simpler, more interpretable model.

In the naive method you described, the impact of a feature is neutralised by setting it to the population mean. You are absolutely right, that this is not an appropriate method, as the prediction of the mean value is not probably not the mean prediction. Also, it does not reflect the feature distribution and does not work for categorical attributes.

Robnik-Sikonja and Kononenko [1] addressed this problem. Their basic idea is the same: the prediction difference between the unchanged instance, and an instance with a neutralised feature. However, instead of taking the mean value to get rid of the features' impact, they create several instance copies, each with a different value. For categorical values, they iterate over all possible categories; for numerical values, they discretise the data into bins. The decomposed instances are weighted by the feature value frequency in the data. Missing data can be ignored by using classifiers that can handle it, or imputing it, e.g. by setting the values to the mean. Conditional importance has been addressed in a second publication by Strumbelj et al [2]. They extended the original approach by not only creating decomposed instances of a single feature, but observed how the prediction changes for each subset of the power set of feature values. This is of course computationally very expensive (as they mention themselves and tried to improve by smarter sampling in Strumbelj and Kononenko [3]).

By the way: for binary data, this problem becomes much easier, as you just have to compare the prediction between attribute is present and not present. Martens and Provost [4] discussed this for document classification.

Another approach for finding groups of meaningful features has been proposed by Andreas Henelius in [5] and [6]. The idea of his GoldenEye algorithm is to permute the data within-class and feature group. Imagine a data table where each row represents an instance and each column is a feature. In each column, all rows that share the same class are permuted. Features are grouped, i.e. permuted together. If the classification on the permuted data is very different (worse) than the original data, the current grouping did not reflect the true grouping. Check out the publications, its better described there. This approach becomes computationally expensive, too.

I'd also like to refer to the publications by Josua Krause [7], [8]. He developed interactive visual analytics workflows for the analysis of binary instance-based classification problems, including an enhanced PDP. They are well-written and an interesting read.

---

[1] Robnik-Šikonja, M. (2004, September). Improving random forests. In European conference on machine learning (pp. 359-370). Springer, Berlin, Heidelberg.

[2] Štrumbelj, E., Kononenko, I., & Šikonja, M. R. (2009). Explaining instance classifications with interactions of subsets of feature values. Data & Knowledge Engineering, 68(10), 886-904.

[3] Štrumbelj, E., & Kononenko, I. (2014). Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 41(3), 647-665.

[4] Martens, D., & Provost, F. (2013). Explaining data-driven document classifications.

[5] Henelius, A., Puolamäki, K., Boström, H., Asker, L., & Papapetrou, P. (2014). A peek into the black box: exploring classifiers by randomization. Data mining and knowledge discovery, 28(5-6), 1503-1529.#

[6] Henelius, A., Puolamäki, K., Karlsson, I., Zhao, J., Asker, L., Boström, H., & Papapetrou, P. (2015, April). Goldeneye++: A closer look into the black box. In International Symposium on Statistical Learning and Data Sciences (pp. 96-105). Springer, Cham.

[7] Krause, J., Perer, A., & Ng, K. (2016, May). Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 5686-5697). ACM.

[8] Krause, J., Dasgupta, A., Swartz, J., Aphinyanaphongs, Y., & Bertini, E. (2017). A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations. arXiv preprint arXiv:1705.01968.

Answer 2

Two other methods worth mentioning here are:

1) Lundberg & Lee's SHAP algorithm, an extension of Štrumbelj & Kononenko's game theoretic approach that they claim unifies LIME and a number of other local importance measures; and

2) Wachter et al.'s counterfactual method, based on generative adversarial networks.

Both methods have advantages and disadvantages. SHAP is very fast and comes with a user-friendly Python implementation. Unfortunately, however, it always compares points against the data centroid, which may not be the relevant contrast in some cases. Also, like LIME and a number of other algorithms, it assumes (or enforces) local linearity, which can lead to unstable or uninformative results when our case of interest is near a distinctly nonlinear region of the decision boundary or regression surface.

Wachter et al.'s solution is more flexible in this regard, a refreshing deviation from what Lundberg & Lee call the "additive feature attribution" paradigm. I'm unaware of any open source implementation, however. The added overhead of GAN training can also be extremely onerous for some datasets.