Interpret predictions of black box models

Question

I am using currently random forest and SVM for a binary classification problem. Especially with random forest it's easy to get the importance of all variables.

But is it also possible to get the relevance for each variable in individual predictions?

I don't need a detailed rule how the result was calculated, but which variable to look at would be very useful for example when using the model for fraud prediction or predictions of failures.

Answer 1

Ribeiro's "Why should I trust you?" paper and blog post provide a method of interpreting black-box models

• paper https://arxiv.org/abs/1602.04938
• blog-post https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime

The model is called "LIME": locally interpretable model-agnostic explanations.

The way it works is to:

• create a set of 'interpretable' features, which may or not be the original input features: could also be a mapping between the two
  • for example, for images, the 'interpretable features' could be contiguous patches of pixels, whereas the input features to the black-box model will likely be pixel values
• sample input/interpretable input features near an example one wishes to explain
• fit a simple, probably linear, model locally, to these local samples
• use this local model to obtain an approximation of which features were most important in classifying the example one wishes to explain