10
$\begingroup$

If feature importance is only calculated from the training set according to here, does it mean one should never compute shap values on test set? What would it mean if I compute shap values from test set ?

For instance, if i have the following code, clf_opt is a random forest estimator and the code runs without any error. What do the shap values of X_test represent in this case? (Note that the validation set (not test set) is part of X_train which goes into K fold cross validation)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
...# perform K fold cross validation
explainer = shap.TreeExplainer(clf_opt,X_train) # training set is background data
shap_values = explainer.shap_values(X_test) # test set is foreground data

I am trying to understand what is the right way to compute shap values for random forest. Most online examples just fit random forest on the entire dataset X without train_test_split and compute shap values on the X without the use of background.

I don't know what is the right way to look at shapley values. Should I copmute shapley values only on training set never on test set?

explainer = shap.TreeExplainer(clf_opt,X_train) 
shap_values = explainer.shap_values(X_train) 

OR (without providing background data)

explainer = shap.TreeExplainer(clf_opt) 
shap_values = explainer.shap_values(X_train) 

OR

explainer = shap.TreeExplainer(clf_opt, X_train) 
shap_values = explainer.shap_values(X_test) 

OR

explainer = shap.TreeExplainer(clf_opt) 
shap_values = explainer.shap_values(X_test) 

I am very lost. Would really appreciate some guidance here.

$\endgroup$

1 Answer 1

7
$\begingroup$

Contrary to standard feature importance, calculating SHAP values on a hold-out set does not make a big difference because ultimately we expect our model to have similar behaviour for instances in the training set, the test set or the validation set. We do not evaluate generalisation; taking an edge case of using a simple linear model, the coefficients $\beta_j$ associated with the feature $x_j$ will have the same impact on the $i$-th instance irrespective of where $\beta_j$ is learned. (This does not mean that the realised effect of feature $x_j$ is exactly the same for all instances; especially for non-linear models we will have different local behaviour and the effect will be conditional to the values of other features.)

As such, using the training set for something like SHAP will allow a post-hoc explainer like SHAP to use a "larger set" and therefore have a more faithful representation. It goes without saying that performance metrics have to be evaluated on a hold-out set.

This is the reason that most tutorials, even in the official SHAP package the section Introduction to Shapley values, use the whole dataset. Ideally, we would use a hold-out set just to ensure no data reuse and potential over-fitting but realistically the SHAP methodology provides in-sample explanations anyway. A fun paper on how this can backfire (in a rather artificially adversarial setting) is Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods (2020) by Slack et al. It shows how in some edge cases SHAP can be misled; even there though we need a specifically trained test-set generator to "muddle" the SHAP predictions, rather than use a standard "sample split".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.