Model Explanation

Question

I have a problem where I am trying to predict something from historic tabular data. We have features {X1,X2,..Xn} and I have a prediction {Y}

I am trying to find a way to explain why the model came up with a prediction. In other words, which features were impacting the most that specific output.

I know I can use algorithms such as Feature Importance or Elastic Net to know which features were more important to the model in general (X features), but I want to know given a single specific input what is more important.

Example: I have a client that wants to predict the budget of certain work that they do. I input some variables they give me and my trained model (let's say a DT or Random Forest regressors) comes up with a prediction value. I know that for the model I have X features that are more important but I can't tell my client every time they ask me to come up with a budget that the budget is for the same reasons(X features) than all of the other works.

I know I can plot the Decision Logic of a decision tree regressor with Graphviz for example, but that's okay only for me to understand. My client would want to see a more user friendly way of looking at the reasons behind that prediction. The final tool should be something like they being able to enter the features in fields, then the budget prediction would come up with lets say percentage circles of impact of their input in the prediction.

Do you know about any ways this type of tool can be implemented at least for experiments?

Do you know about any ways of displaying the specific decision path followed by a decision tree regressor of only one prediction?

Answer 1

Machine learning explainability is an area of active research, quite popular in recent years. This question cannot be answered briefly, because there are many research papers, tutorials, and even books on this subject. It would be an impossible task to summarize them in few sentences. I would encourage you to check the freely available online book Interpretable Machine Learning by Christoph Molnar that deals with this subject.

TL;DR for model-agnostic explanations of individual predictions, you can check methods such as SHAP or LIME. However, keep in mind that those methods approximate your model to give you the explanations, so it is always possible that those explanations may be incorrect.

Answer 2

To answer your question from a totally uncorrelated perspective, it's instructive to note that in financial engineering and economics, there needs to be some kind of uncertainty around cost or budget estimates, since a single prediction based on the best features won't be sufficient. Even if you employ ML and determine which features are the most informative through model-building strategies using univariate/multivariate regression models, importance scores from random forests, or any other feature selection method, you still won't have anything to work with that explains how uncertain your cost estimate is. What is needed is Monte Carlo cost estimation.

What could be more helpful would be to identify the best features which predict cost, develop an objective function, e.g. cost = hourly labor rate * #employees * hours of work. Then generate histograms for these 3 example inputs, determine which probability distributions fit the best, then simulate quantiles from each, each time calculating the cost. You will then have a histogram of cost, which you give to the customer, along with providing the 10th and 90th percentile values of cost (or 5th, 95th). If you were designing an oil rig and presenting cost estimates for a loan from a bank on an enterprise scale, this is what you would be doing. Below is a simple example of a run for a rollout of a new product after manufacturing.

The cost uncertainty distribution is on the lower right of the panel.