Finding most similar training samples for a given ML model output

Question

Let's say you have trained a regression model. Now, in production, you get a new input, and you want to identify K samples in the training set that are the most similar to the new input as far as the model is concerned.

I do realize that the solution to this depends entirely on the similarity metric, and that the appropriate similarity metric is ill-defined - i.e., how do we know that our similarity metric is doing a good job? But we probably could tell it is doing a good job qualitatively, i.e. know-it-when-we-see-it.

But let us start from a simple example. Let's say our model is a linear regression, y = w0*1 + w1*x1 + w2*x2 + w3*x3. It seems that considering the distance between [w0*1, w1*x1, w2*x2, w3*x3] for the two samples would be a good idea:

1. It normalizes the features according to feature importances
2. "Similar" inputs would produce similar outputs

It certainly seems better than measuring the distance between [x1, x2, x3] or [normalized(x1), normalized(x2), normalized(x3)], as these do not take into account any information learned by the model, apart from maybe feature importance. It also seems better than simply comparing [y1] with [y2] - samples in very different parts of the feature space can have similar outputs, but it does not seem right to say that the samples are "similar", i.e. the model gave them similar outputs for similar reasons.

Now let's say we have a more complex example, where our model is XGBoost. We could follow a similar approach - for every sample, break down the final model output into a sum of contributions from each feature (e.g. using the approach used by eli5.explain_prediction()), and then check the distance between vectors of such contributions for the new input and training set samples.

1. Does the approach seem sensible?
2. Am I reinventing the wheel here, i.e. are there any known existing approaches for interpreting ML models by finding most similar training set samples according to the model?

Answer 1

Found a recent paper that presents a similar approach to the one I described in the question: "Consistent Individualized Feature Attribution for Tree Ensembles", Lundberg et.al., KDD 2018.

They use Shapley values to explain the model prediction for each sample as a sum of contributions from each feature, and then compare the distances between the resulting vectors to find samples that are similar according to the model. The results seem to be fairly intuitive and interpretable:

So the approach in the question appears to be both sensible and used elsewhere.

Answer 2

If you are looking to pick 'n' closest samples from the training data set that is similar to an out-of-sample observation, why not simply use the features you have to get a distance with each training observation - euclidean, manhattan or whatever (depends on the feature types)? The training sample with the smallest distance to the new observation would be the most similar.