I am a deep learning researcher that would like to use SHAP values to assess the relative importance of input features on the model's final score. Colleagues of mine have taken issue with the method outright. The rub is that the method replaces absent input values (due to coalitions in between none and all features) with values selected from random instances of background data. This potentially creates implausible instances due to conditional dependence of input features, and in my specific case would be e.g. protein sequences that would never occur with certain charges, or unphysical protein sequences themselves. It seems like there are few, if any, datasets where all input features are independent, so this issue is simply inevitable with the method. Furthermore conditional sampling would be very hard to implement in my case, as there would be many many rules to follow.
I am interested to hear opinions of people who are familiar with the theory or implementation of SHAP values. Does the issue I outlined invalidate SHAP on datasets with conditional feature dependencies? How tolerable is the issue of sampling implausible inputs? How do you assess the appropriateness of this method for explaining black box models?
Thanks in advance
