1
$\begingroup$

How is the contribution of features extract from explaining a regression model with LIME locally related to the predicted output of the surrogate model?

I thought that LIME is additive (some blog post as source), but wasn't able to get this additiveness in my example. I'll illustrate my tries:

I explain a Random Forest model via LIME using:

# Create explainer for the Random Forest model
rf_explainer = lime.lime_tabular.LimeTabularExplainer(rf_X_train.values, feature_names=config['rf_features'], class_names=['duration'], mode='regression')

# Explain values
rf_exp = rf_explainer.explain_instance(rf_X_sc.values[0], rf_regressor.predict)
feature_importance = [x[1] for x in sorted(rf_exp.__dict__['local_exp'][0], key=lambda tup: tup[0])]
y_surrogate = rf_exp.__dict__['local_pred'][0]
result = [scenario, 'rf', rf_y_sc.iloc[0], rf_prediction[0], y_surrogate] + feature_importance

The result is:

scenario     sc_1
method       rf
y_real       390.0
y_rf         312.910836
y_surrogate 1846.915013   # <- Surrogate seems to be pretty bad
feature_0    -21.599091
feature_1    -27.415115
feature_2     13.378463
feature_3   -199.55607
feature_4     -7.411741
feature_5   -194.997414
feature_6     -5.433271
feature_7   -334.37682
feature_8    -19.342806

Because LIME uses a linear model as a surrogate, I expected that

-21.599091 + -27.415115 + 13.378463 + -199.55607 + -7.411741 + -194.997414 + -5.433271 + -334.37682 + -19.342806

is y_surrogate. Unfortunately, the sum is -796.753865, which is unequal to 1846.915013.

My next thought was, that I have to subtract -796.753865 from 1846.915013 to come to something like the base value of the surrogate model; but I checked with a second sample that uses the same explainer - it showed a different value.

What am I understanding/doing wrong? Thanks for your help!

Edit: Here is an example that you can easily reproduce - the behavior is the same.

 from sklearn.datasets import load_iris
 from sklearn.model_selection import train_test_split
 import xgboost as xgb
 import lime
 import lime.lime_tabular     
 

 def main():
     random_state = 42
     df = load_iris(as_frame=True)['data']     

     # XGBoost regressor that predicts 'sepal length (cm)'
     target = ['sepal length (cm)']
     features = ['sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
     X_train, X_test, y_train, y_test = \
         train_test_split(df[features], df[target], test_size=0.33, random_state=random_state)     

     regressor = xgb.XGBRegressor(n_estimators=30, random_state=random_state)
     regressor.fit(X_train, y_train)     

     sample_to_explain = X_test.iloc[0:1,:]
     y_test_p = regressor.predict(sample_to_explain)  # Predict first sample in X_test
     xgb_explainer = lime.lime_tabular.LimeTabularExplainer(X_train.values, feature_names=features,
                                                            class_names=target, verbose=True, mode='regression')
     exp = xgb_explainer.explain_instance(sample_to_explain.values[0], regressor.predict)
     feature_importance = [x[1] for x in sorted(exp.__dict__['local_exp'][0], key=lambda tup: tup[0])]
     y_surrogate = exp.__dict__['local_pred'][0]
     print(f'xgb,\n'
           f'Real y: {y_test.iloc[0].values[0]},\n'  # 6.1
           f'Predicted by XGBoost: {y_test_p},\n'  # 6.2832994
           f'Predicted by LIME surrogate: {y_surrogate},\n'  # 6.127684969038174
           f'Feature importances: {feature_importance},\n'  # [0.03934168590785398, -0.5762293534314864, -0.08231280097169107]
           f'Values from sample: {sample_to_explain.values[0]}\n')  # [2.8 4.7 1.2]     

     # 2.8*0.03934168590785398 + 4.7 * -0.5762293534314864 + 1.2 * -0.08231280097169107 = -2.6968966017520244 != 6.127684969038174     
 

 if __name__ == '__main__':
     main()
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The numbers you get out from LIME will follow a linear model such that:

\begin{equation} y_{surrogate} = x_0+f_1x_1+...+f_nx_n \end{equation}

The equation you have in your question is:

\begin{equation} y_{surrogate} = f_1+...+f_n \end{equation}

I can't say why the $y_{surrogate}$ value is so bad without knowing more about your problem though. It seems like it might be a bug in the code.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Previously, I also tried that, but the result was even worse: 6.000000 * -21.599091 + 25.000000 * -27.415115 + 1.000000 * 13.378463 + 119.000000 * -199.55607 + 17.000000 * -7.411741 + -73.943733 * -194.997414 + 40.817074 * -5.433271 + -73.949699 * -334.37682 + 40.822285 * -19.342806 = 13459.748259382872, so, I didn't even wrote the result in the question. The explanations generated make sense and the model seems to have learned the right thing, I checked that with around 120 other samples. The model predicts seconds and the mean absolute error of the random forest model is around 185s. $\endgroup$
    – So S
    Commented Sep 28, 2021 at 11:43
  • $\begingroup$ I think the first step would be to debug why the $y_{surrogate}$ is so bad. It's hard to say why as I can't access any of the inputs of the line which generate that quantity. That's where I would start though. $\endgroup$
    – Adam Kells
    Commented Sep 28, 2021 at 13:14
  • $\begingroup$ Thanks! I built some reproducible results and shared the source code in the original question. The behavior is the same. So most likely, I have the same error in both blocks of code. $\endgroup$
    – So S
    Commented Sep 28, 2021 at 14:09
  • $\begingroup$ Okay that's great, at least in this example, $y_{surrogate}$ is close to the expected value. I made an edit to my answer as I realised the linear fit will also have some intercept coefficient. As far as I can tell, this number isn't output by LIME. $\endgroup$
    – Adam Kells
    Commented Sep 28, 2021 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.