I'm looking for a reliable way of explaining decisions of a deep neural network classification model. More specific, I want to know, which features contributed the most to a decision for a particular observation. For instance, imagine a credit scoring system - you would probably like to know why your model accepts or rejects an application.

Unfortunately, I did some research and only found various methods (e.g. Grad-CAM) for Convolutional Neural Networks in the field of computer vision. I'm also aware of methods like LIME that are meant to explain any classifier's decisions. But, as far as I understand the paper, the problem is that LIME doesn't really work for relatively complicated models / data sets and there is a set of assumptions one have to satisfy.

I'd like to find a method for other architectures (e.g. LSTM, Auto-Encoders, Dense NNs) and problems (e.g. sentiment analysis). Also, I implement my models in Python (Tensorflow, Keras, PyTorch). Is there any Python library that could help?

  • 1
    $\begingroup$ Could you point us to what in the LIME paper says that it "doesn't really work for relatively complicated models / data sets"? $\endgroup$
    – Pop
    Commented Jun 25, 2018 at 11:16

1 Answer 1


SHAP is the closest to what you need: it uses the same framework as LIME. The authors showed in their paper (https://arxiv.org/pdf/1705.07874.pdf) that

  • several recently proposed methods (including LIME) belong to the same class
  • among this class of methods, only one (SHAP) has som desirable properties (local accuracy, "missingness", consistency; see their paper for more details)

So that, if you have already looked into LIME, SHAP will look familiar to you.

Here is the associated Python code developed by the authors https://github.co/slundberg/shap

I encourage you to use this technique; although I have not heard of limitations relatively to "complicated models / data sets" as you stated...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.