Understanding functioning of LIME(Local Interpretable Model-agnostic Explanations)

Question

Following are the steps that occur in LIME's algorithm

https://cran.r-project.org/web/packages/lime/vignettes/Understanding_lime.html

I have been trying to read and understand why this process is followed. My questions are around some of steps.

1) Permutation is done based on the distributions of the input variables from actual data. Hence, fake data is created. Question - why do we need to create fake data? if we need observations around the local region of the observation we are predicting for, can't we just take it from the actual data using a distance metric? followed by building simple model on this small subset of the data

2) How exactly do we use the similarity score here from the fake data(simulations) and actual data?

3) In step 6, Fit a simple model to the permuted data, explaining the complex model outcome with the m features from the permuted data weighted by its similarity to the original observation what exactly are we modelling after selecting best 'm' variables?

How would the simulations turn out if i built a model on say a big data set of 100000k rows and 12 cols, however for prediction i have just one row. Will simulation replicate the same row multiple times and then go and get the similar rows from actual big dataset, build a model on that or simulated data set and then derive coefficients?

Answer 1

When using LIME to explain an instance $i$, we create fake data because through them and their subsequent labelling we can infer how our complex learner $L$ would behave in neighbourhood of $i$ through a simpler learner. (See comment in the end.) What you describe about taking the actual data is perfectly reasonable given that the reasonable neighbours exist and we can "look around" the instance $i$. If an instance $i$ does not have close neighbours it become difficult to explain why a particular result was obtain because we lack a comparable sample. The perturbation (in theory) ensures that we have a reasonable chance of creating close neighbours and thus investigate $L$'s behaviour on them.

Regarding on how the similarity score is used and what we model exactly: The similarity score is used as a weight. Observations that are close (in terms of similarity score) to the instance $i$ are given higher weight, i.e. we are saying to the simple/explainer model $S$ to treat these (artificial) points as more significant.To that extent, given that usually we use some kind of penalised regression (e.g. LASSO) for our explainer $S$ we readily get the "$m$ more explanatory" variables. Also, as standard regression procedures can be easily extended to weighted regression routines we thus get a straightforward methodology! To be clear: the outcome that we model with $S$ is the artificial labels we assigned to the "fake data" through our learner $L$.

Finally, in regards to using a very long (high $N$) but relatively thin (small $p$) dataset. (i.e. we are in a $N >> p$ situation) That is perfectly fine in terms of LIME and suggests that is is more likely that we have to have relevant neighbours when training our original learner $L$. Do note that LIME is not strongly affect by the number of available "true" training examples as ultimately the explainer is trained on the "fake data". The learner $L$ will indeed benefit from a large (and potentially representative) sample but that is not directly affecting LIME's computation aside the fact that the labelling of the fake data will be (hopefully) more representative.

Comment: While almost all implementations employ some notion of data perturbation the validity of this approach to generate new data is not ensured. For example, assuming that we have a dataset where we look at yearly earnings as the response variable and some of explanatory variables are the physical age and the year of work experience of a person; simple perturbation does not stop us from having "fake data" where a 20 year old person has 21 years of work experience. This clearly nonsensical and pretty uninformative but simple approaches might fall into this trap.