8
$\begingroup$

Background

I'm validating a model and as part of the process I've been calculating SHAP values for different validation datasets.

I've calculated SHAP values for every sample in each dataset taken their absolute values, summed, and normalised them. This allows me to see their aggregate contribution to the model predictions, as a percentage, for each dataset.

Although my top 20 features remain the same, I've noticed there are minor ranking shifts in their percentage contributions.

I believe this is caused by the over-emphasis or under-emphasis of patterns that the model has been trained to pick up, and which use specific sets of features. However, I'm not sure if this is a reasonable interpretation.

Question

What causes shifts in the contribution of features, as measured by SHAP, on different datasets for the same model?

$\endgroup$
2
  • 1
    $\begingroup$ Take a look at this page, which I think might answer your question. If predictors are correlated (as they almost always are, in practice), different samples will tend to show different relative feature importance as one or another of the correlated predictors ends up more highly associated with outcome by chance. $\endgroup$
    – EdM
    Nov 30, 2023 at 16:53
  • 1
    $\begingroup$ SHAP importance rankings can also change when all features are independent. This is just a form of sampling uncertainty. $\endgroup$
    – Michael M
    Nov 30, 2023 at 17:07

1 Answer 1

11
$\begingroup$

Why can a model's SHAP values change on a new dataset?

WHY SHOULDN'T THEY?

When you calculate a mean on a new data set, you expect to get a slightly (or radically) different value.

When you calculate a variance on a new data set, you expect to get a slightly (or radically) different value.

When you calculate a regression coefficient on a new data set, you expect to get a slightly (or radically) different value.

There is variability to measures of feature importance just like there is variability to other statistics of interest. After all, the SHAP value (for example) is a function of random variables and, thus, a random variable itself. You calculate the SHAP value from some new numbers. I would expect the output to differ at least a bit.

What amazes me is how little attention this gets. Many people seem to be willing to take measure of feature importance (such as SHAP) as absolute truth, when they would demand some kind of uncertainty quantificataion (e.g., test statistic, p-value, confidence interval, credible interval, etc) from a mean or a regression coefficient. As you see from your work, there is variability to the SHAP values and ranks of feature importance.

In this linked video of a keynote presentation, Vanderbilt's Frank Harrell basically dares practitioners to calculate bootstrap confidence intervals of their measures of feature importance, with a remark that if such a practitioner is afraid to calculate that confidence interval, then they shouldn't be using the technique they're using.

$\endgroup$
5
  • 1
    $\begingroup$ THANK YOU DAVE! I do expect SHAP values to change, particularly because I'm using decision trees and I reason different distributions will follow different paths down each tree hitting different features on the way. Perhaps the question under the question is, is this a reasonable interpretation of what's happening? Thank you for the bootstrapping video, I'll watch it tonight! $\endgroup$
    – Connor
    Nov 30, 2023 at 19:44
  • 3
    $\begingroup$ @Connor That interpretation sounds reasonable. As you change the data, the exact path from features to outcomes changes (at least a bit), leading to the features having changed importance, possibly changed ranks of importance. $//$ Harrell argues in the linked video that these changes are quite dramatic. $\endgroup$
    – Dave
    Nov 30, 2023 at 19:46
  • $\begingroup$ Adding my two cents to the video: (1) The OP does not (probably) use the importance values to select features, just to study their importance. So there is less pressure to not make a mistake. (2) Bootstrapping the full model process (including model-tuning, model fitting) is relatively costly in many situations. Just Boostrapping TreeSHAP is cheap, but that would give much too optimistic results. $\endgroup$
    – Michael M
    Nov 30, 2023 at 21:16
  • 1
    $\begingroup$ With the answer accepted, I am inclined not to make a modification. However, I wonder if the “why” should be removed from my bold text. @Connor any thoughts? $\endgroup$
    – Dave
    Dec 1, 2023 at 2:02
  • $\begingroup$ @Dave I think your answer is very good, and I like the bold "why shouldn't they?". It drives the point home straight away. If you decide to make some additions, I would appreciate more insight into why they might change for decision trees in particular, but also neural nets. My personal belief is that different data distributions emphasise different patterns, but if there's a more rigorous way of saying that, I think that would be a great addition! $\endgroup$
    – Connor
    Dec 1, 2023 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.