How to extract global feature importances of a black box model from local explanations with LIME?

The LIME framework (see https://arxiv.org/abs/1602.04938 and https://github.com/marcotcr/lime) allows for per-instance explanations of a black box model's predictions. It does so by random sampling around a sample of interest in feature space and fitting a linear model to those data points that is easily interpretable.

LIME can then output an "explanation" for the model's prediction concerning an individual instance. This explanation consists of "weights" which should be interpreted "by applying them to the prediction probabilities". (See the LIME GitHub page for examples.) In my understanding, to put it simply: The greater the weight of a feature, the more relevant this feature is for the particular decision.

My question is whether these weights can be somehow used to make statements about some model characteristics on a global level (in contrast to the local explanations). Specifically, I want to know if I can generate some kind of feature importance plot like in the case of tree ensembles.

The approach I am thinking of is to go through a number of samples of one class and generate LIME explanations (i.e., the weights) for each one of them. Then, I would just sum up all the weights for each feature and eventually use this data to create a simple bar plot, similar to the common feature importance plots mentioned above for tree-ensemble classifiers. Normalization of the resulting feature importance distribution is of course a different story, but if I just want to know which features are most important most of the time, I am only interested in the relative importances of the features anyway.

Here is the code implementation of my proposed approach for the case of binary classification using a trained classifier (model). (Please excuse any clumsiness in my code as I'm quite new to Python.)

import lime
import lime.lime_tabular

explainer = lime.lime_tabular.LimeTabularExplainer(X,
feature_names=labels,
class_names=['background','signal'],
discretize_continuous=False)

# create array with predicted signal probabilities
proba_signal = model.predict_proba(X)[:,1]

importances = { i: 0.0 for i in range(X.shape[1]) }

# number of instances to generate explanations for
num_explain = 1000

for i in range(num_explain):

# collect feature importances for the signal class only
if proba_signal[i] > 0.5:

exp = explainer.explain_instance(X[i],
model.predict_proba,
num_features=X.shape[1])
exp_map = exp.as_map()

# get all feature labels of class "1"
feat = [exp_map[1][m][0] for m in range(len(exp_map[1]))]

# get all feature weights of class "1"
weight = [exp_map[1][m][1] for m in range(len(exp_map[1]))]

# sum the weights, for each feature individually
for m in range(len(feat)):
importances[feat[m]] = importances[feat[m]] + weight[m]

# normalize the distribution (probably meaningless...)
for i in range(X.shape[1]):
importances[i] = importances[i] / (num_explain*1.0)


Plotting the summed (and sorted) weights for each feature results in the following plot. The horizontal axis contains the features, encoded by integer labels.

Is this a valid approach for generating feature importances for the global model scope from local LIME explanations?

Thanks!

• To your final question, the answer is no. The entire point of this work is that global explanations are difficult to find. Local explanations can be useful despite not necessarily saying anything about global explanations (esp. for complex models). From the article itself: "features that are globally important may not be important in the local context, and vice versa." – galoosh33 Dec 7 '17 at 9:46
• Yes, I get this argument and that LIME is an approach for local explanations and not global ones. I don't completely trust my approach above either, to be honest, that's why I asked. However, I'm not yet fully convinced that I'm wrong. Particularly, I don't get why my approach might no work for binary classification when I want to get some kind of general feeling for how often particular features were important for individual decisions. If one feature has a large (absolute) weight for the majority of instances, I can indeed consider it a globally important feature, can't I? – tempse Dec 7 '17 at 14:41