I am using currently random forest and SVM for a binary classification problem. Especially with random forest it's easy to get the importance of all variables.

But is it also possible to get the relevance for each variable in individual predictions?

I don't need a detailed rule how the result was calculated, but which variable to look at would be very useful for example when using the model for fraud prediction or predictions of failures.


Ribeiro's "Why should I trust you?" paper and blog post provide a method of interpreting black-box models

The model is called "LIME": locally interpretable model-agnostic explanations.

The way it works is to:

  • create a set of 'interpretable' features, which may or not be the original input features: could also be a mapping between the two
    • for example, for images, the 'interpretable features' could be contiguous patches of pixels, whereas the input features to the black-box model will likely be pixel values
  • sample input/interpretable input features near an example one wishes to explain
  • fit a simple, probably linear, model locally, to these local samples
  • use this local model to obtain an approximation of which features were most important in classifying the example one wishes to explain
  • $\begingroup$ Thanks, I found this LIME paper but was wondering if it works only with images and texts as they were using it. In the fraud detection case, I am not sure how I should define "near the example", do I need a distance function then to find examples close to my prediction case? $\endgroup$ – MikeHuber Mar 27 '17 at 14:44
  • $\begingroup$ Another question I have is, when I generate the new samples by randomizing some of them, how do I define the label then for those cases? $\endgroup$ – MikeHuber Mar 27 '17 at 15:26
  • $\begingroup$ @MikeHuber - neighbors of the example are generated by perturbing the features assuming normal distribution. The labels of the neighbors are generated by the black-box model - they are the predictions for each neighbor $\endgroup$ – ihadanny Apr 27 '17 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.