# Understanding the output of LIME - How is the contribution of features related to the predicted output?

How is the contribution of features extract from explaining a regression model with LIME locally related to the predicted output of the surrogate model?

I thought that LIME is additive (some blog post as source), but wasn't able to get this additiveness in my example. I'll illustrate my tries:

I explain a Random Forest model via LIME using:

# Create explainer for the Random Forest model
rf_explainer = lime.lime_tabular.LimeTabularExplainer(rf_X_train.values, feature_names=config['rf_features'], class_names=['duration'], mode='regression')

# Explain values
rf_exp = rf_explainer.explain_instance(rf_X_sc.values, rf_regressor.predict)
feature_importance = [x for x in sorted(rf_exp.__dict__['local_exp'], key=lambda tup: tup)]
y_surrogate = rf_exp.__dict__['local_pred']
result = [scenario, 'rf', rf_y_sc.iloc, rf_prediction, y_surrogate] + feature_importance


The result is:

scenario     sc_1
method       rf
y_real       390.0
y_rf         312.910836
y_surrogate 1846.915013   # <- Surrogate seems to be pretty bad
feature_0    -21.599091
feature_1    -27.415115
feature_2     13.378463
feature_3   -199.55607
feature_4     -7.411741
feature_5   -194.997414
feature_6     -5.433271
feature_7   -334.37682
feature_8    -19.342806


Because LIME uses a linear model as a surrogate, I expected that

-21.599091 + -27.415115 + 13.378463 + -199.55607 + -7.411741 + -194.997414 + -5.433271 + -334.37682 + -19.342806


is y_surrogate. Unfortunately, the sum is -796.753865, which is unequal to 1846.915013.

My next thought was, that I have to subtract -796.753865 from 1846.915013 to come to something like the base value of the surrogate model; but I checked with a second sample that uses the same explainer - it showed a different value.

What am I understanding/doing wrong? Thanks for your help!

Edit: Here is an example that you can easily reproduce - the behavior is the same.

 from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import xgboost as xgb
import lime
import lime.lime_tabular

def main():
random_state = 42

# XGBoost regressor that predicts 'sepal length (cm)'
target = ['sepal length (cm)']
features = ['sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
X_train, X_test, y_train, y_test = \
train_test_split(df[features], df[target], test_size=0.33, random_state=random_state)

regressor = xgb.XGBRegressor(n_estimators=30, random_state=random_state)
regressor.fit(X_train, y_train)

sample_to_explain = X_test.iloc[0:1,:]
y_test_p = regressor.predict(sample_to_explain)  # Predict first sample in X_test
xgb_explainer = lime.lime_tabular.LimeTabularExplainer(X_train.values, feature_names=features,
class_names=target, verbose=True, mode='regression')
exp = xgb_explainer.explain_instance(sample_to_explain.values, regressor.predict)
feature_importance = [x for x in sorted(exp.__dict__['local_exp'], key=lambda tup: tup)]
y_surrogate = exp.__dict__['local_pred']
print(f'xgb,\n'
f'Real y: {y_test.iloc.values},\n'  # 6.1
f'Predicted by XGBoost: {y_test_p},\n'  # 6.2832994
f'Predicted by LIME surrogate: {y_surrogate},\n'  # 6.127684969038174
f'Feature importances: {feature_importance},\n'  # [0.03934168590785398, -0.5762293534314864, -0.08231280097169107]
f'Values from sample: {sample_to_explain.values}\n')  # [2.8 4.7 1.2]

# 2.8*0.03934168590785398 + 4.7 * -0.5762293534314864 + 1.2 * -0.08231280097169107 = -2.6968966017520244 != 6.127684969038174

if __name__ == '__main__':
main()


The numbers you get out from LIME will follow a linear model such that:

$$\begin{equation} y_{surrogate} = x_0+f_1x_1+...+f_nx_n \end{equation}$$

The equation you have in your question is:

$$\begin{equation} y_{surrogate} = f_1+...+f_n \end{equation}$$

I can't say why the $$y_{surrogate}$$ value is so bad without knowing more about your problem though. It seems like it might be a bug in the code.

• Previously, I also tried that, but the result was even worse: 6.000000 * -21.599091 + 25.000000 * -27.415115 + 1.000000 * 13.378463 + 119.000000 * -199.55607 + 17.000000 * -7.411741 + -73.943733 * -194.997414 + 40.817074 * -5.433271 + -73.949699 * -334.37682 + 40.822285 * -19.342806 = 13459.748259382872, so, I didn't even wrote the result in the question. The explanations generated make sense and the model seems to have learned the right thing, I checked that with around 120 other samples. The model predicts seconds and the mean absolute error of the random forest model is around 185s.
– So S
Sep 28, 2021 at 11:43
• I think the first step would be to debug why the $y_{surrogate}$ is so bad. It's hard to say why as I can't access any of the inputs of the line which generate that quantity. That's where I would start though. Sep 28, 2021 at 13:14
• Thanks! I built some reproducible results and shared the source code in the original question. The behavior is the same. So most likely, I have the same error in both blocks of code.
– So S
Sep 28, 2021 at 14:09
• Okay that's great, at least in this example, $y_{surrogate}$ is close to the expected value. I made an edit to my answer as I realised the linear fit will also have some intercept coefficient. As far as I can tell, this number isn't output by LIME. Sep 28, 2021 at 14:37