1
$\begingroup$

Lime explanatition as example

On the picture above you can see prediction probabilities. In this case it shows 100% poisonous. However, on next two figures it shows that there is actually small probability of other class (with feature gill_size, blue-colored). What is the reason behind this inconsistency?

Picture from http://pythondata.com/local-interpretable-model-agnostic-explanations-lime-python/

$\endgroup$

1 Answer 1

3
$\begingroup$

There is no inconsistency. The first (left-most) numbers reflect the predictions of the classifier used (here this classifier is a random forest (*)). The second (central) numbers reflect the average influence of that particular feature value in the final predictions. These two sets of numbers should indeed convey similar information but they do not need to be exactly the same.

In particular, the first numbers come from our (potentially highly sophisticated) full model that encapsulates the associations across all the features in our training sample. The second numbers encapsulate the behaviour of the linear model used by LIME in the neighbourhood of the sample-point we trying to explain. (Notice that LIME usually does not employ all the available features when making an explanation, we usually set the num_features to be smaller than the total number of available features.) For the example shown, the predicted probability of that particular mushroom being poisonous is $1$. The second numbers suggest that, given the training data available, having odor=foul would increase on average the predicted probability of that mushroom being poisonous by $0.26$ and similarly, having gill-size=broad would decrease on average the predicted probability by $0.13$. That does not necessitate that the overall predicted probability from the full classifier is the sum of these average changes - thus there is no inconsistency. :)

(*)While the blog-post link provided does not show the source of that image, the full notebook where this picture is taken from, can be found here. This how I know that the underlying full model is an RF.

$\endgroup$
3
  • $\begingroup$ BTW, welcome to the CV community! $\endgroup$
    – usεr11852
    Sep 23, 2018 at 20:19
  • $\begingroup$ Hi @user11852. Very useful explanation. I have a question though. If all those 2nd number doesn't add up to the total probability, can I know what do they add up to? We don't see the intercept here. So, it is possible that for a class 1, we can still get lot of negative numbers (feature with negative influence in 2nd diagram). However, with high intercept, that class still became positive. How do you interpret Lime in that case? $\endgroup$
    – The Great
    Feb 15, 2022 at 14:33
  • $\begingroup$ Assuming we use logistic regression as the underlying explainer we would interpret LIME coefficient as affecting the log-odds. (I saw your question I will try to answer it tonight or tomorrow night) $\endgroup$
    – usεr11852
    Feb 15, 2022 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.