Interpreting/Quantifying what is causing changes in ML model predictions week over week Ask

Question

We are currently predicting an online students likelihood of completing a class each week. We use a lot of demographic information (which is constant throughout the class), as well as a small number of performance metrics (test results, logins, time spent etc). We have been asked to provide some insight each week for any significant changes in the prediction. I.e. if student X goes from 0.8 probability to complete the course to 0.2 they want to be able to see why.

Currently we are testing out Randomforest, XGBoost and Logistic regression models. I have been trying to use the LIME package for this, but the issue is the coefficients/weights change for many of the demographic features which are static. So from week 4 to week5, the Lime explanation will weigh something like Age, or incoming GPA slightly differently, which is harder to interpret.

The change in prediction should be due to the performance metrics because they are the only thing that is changing, so the route I am taking now is to multiply the feature delta with the feature weights given by lime for the previous week's prediction.

There are basically two main forces at play that change week over week - one is how far in to the course they are, and the other are the performance metrics. Those are the only things that can change. So Lime tells me how impactful the features are relative each other for that prediction. Then I apply this to whatever has changed week over week to approxiate what had the biggest impact.

Does this sounds like a good way to approach this or is there a better way?

Another issue is that we have slightly better performance training weekly models instead of one giant model. However that causes me some issue for explainability, as the next weeks model might weigh one of the static features differently and this could very well drive the change in prediction, but doesn't make sense for explaining week over week changes. But the two approaches are close in performance, so right now I'm favoring the single model over the multi/mini model approach as it has better interpretability week over week.

Any inputs/feedback are much appreciated

Answer 1

Interesting approach. I am not sure if it is the best possible though. If you are using random forests there are other measures you might use to get feature relevance (e.g., decrease in accuracy if a feature is removed). This has advantages. It is an easy and conceptually simpler approach. Many complex XAI techniques such as LIME are often not robust. That is, if you analyze fairly similar models, the explanations can vary a lot. You have to check if LIME (or whatever technique you are using) is robust. Furthermore, explainability techniques like LIME serve to understand static models. There are other methods focusing on time aspects.