I'm trying to use LIME to locally explain black-box models predictions. It works by generating 5000 neighbors for an instance by perturbing the features. Then it does
forward-selection by iteratively fitting a simple
Ridge model over that local neighborhood, where the label is the black-box model prediction. After
n iterations, the selected features are considered as the most important for the black-box decision, and the weights are indicative of how important.
I'm measuring how good is that explanation by measuring the
R^2 of the local model.
When the black-box model is
LogisticRegression, the average
R^2 is ~0.65 which I consider pretty good. However, once I switched to a more sophisticated
RandomForest black-box model, I'm getting a very low (~0.06) mean
Why is that? does that mean that the LIME method isn't good for explaining the surface of my model predictions?
Technicalities: I have 6 features -
age, gender, hemoglobin_last, hemoglobin_slope, creatinine_last and creatinine_slope and the outcome I'm trying to predict is chronic_renal_failure. The data is pretty imbalanced: 1.5M controls and 5K cases. For measuring quality I randomly selected 100 cases and 100 controls and averaged the
R^2 score of their explanation.
(related to Interpret predictions of black box models)