As I've been trying to get a grasp on the field of interpretable ML I have encountered many interesting papers and methods, but I lack the perspective to determine their importance to the field as a whole. To those with a better perspective: What are the most important/seminal/popular methods in the field of interpretable ML and what are the 'first principles' that make them so?


I think there are three big categories of interpretation methods:

Surrogate Models

Replace the complex black box model with an interpretable model that approximates the black box model reasonably well. The surrogate model can either explain the black box model as a whole, or parts of it, even at the level of individual predictions. LIME [1] is an implementation of the idea that you replace the black box prediction function with an interpretable model (for example a linear regression model) at the level of an individual data point and its neighborhood.

Feature Effects

Some methods try to summarize the average effect a feature has on the prediction, for example, partial dependence plots (PDP) [2], accumulated local effect plots [3] and individual conditional expectation (ICE) curves. They usually summarize the feature effect averaged over the whole data, except for ICE curves which show the effects on an individual level. I would also see methods like counterfactual explanations here (e.g. "If feature x2 would be increased by 1 point, the predicted probability would increase by 5%") as describing the effect of features for an individual prediction.

Feature Importance

Feature importance ranks the features by how much the model relies on them for making predictions. This can be formulated by how much the feature contributed to the performance of the model [5], but also by how much of the variance of the predictions is explained by a feature.

Further properties

There are some further properties by which you can distinguish methods for interpretable machine learning:

  • Scope: Global vs. Local: Is the interpretation for the entire model? Or maybe just for an individual prediction? For example, LIME produces local explanations while PDP produces global ones.
  • Model-agnostic vs. Model-specific: Does the interpretation method only work for certain models because it relies on the model-specific structure?
  • Type of data: It depends on the data type which interpretation method can be used. PDP doesn't make sense when the inputs are image pixels. I would see tabular, text and image data as three common classes.

[1] https://arxiv.org/abs/1602.04938

[2] https://statweb.stanford.edu/~jhf/ftp/trebst.pdf

[3] https://arxiv.org/abs/1612.08468

[4] https://amstat.tandfonline.com/doi/abs/10.1080/10618600.2014.907095

[5] https://arxiv.org/abs/1801.01489

  • $\begingroup$ Sir! your book is just beautiful. Orthogonal to all mainstream ML books and courses. I loved Chapters 5 and 6, very eye-opening and insightful. Thank you very much! $\endgroup$ – Esmailian Mar 15 at 18:13
  • $\begingroup$ Thank you this outline is very useful! $\endgroup$ – devin132 Mar 15 at 21:29
  • $\begingroup$ Would you mind elaborating on the difference between feature effects and feature importance? $\endgroup$ – devin132 Mar 24 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.