So I know there is a feature_importances_ variable under the XGBoost classifier. I was wondering if there is a way to see the deciding features for each observation? This will allow me to understand why the machine learning algorithm predicted its class for each observation.
I looked into LIME: https://github.com/marcotcr/lime
And it looks to have what I need. I'm just curious if there are any other libraries that do something similar.
Have a look at SHapley Additive exPlanations, which has a game theoretic basis ("How do you 'optimally' assign credit to different players in a team for the team outcome?" - here variables are treated like players). For xgboost and similar algorithms (e.g. LightGBM), the calculations - that involve considering all orders in which you could give features importance - can be done quite efficiently so that this approach has become quite popular. SHAP values have their flaws (like probably most other current explainability approaches): see e.g. this paper or the work discussed in this podcast episode.
More generally, there's a great online book on explainable ML that's worth looking into. Chapters 5.9 and 5.10 deal with SHAP, but there's also chapters on LIME etc., which are of course also well-known approaches.
You should have a look at this great tuto that gives interpretability insight for any model. In particular the package eli5 in Python and this kernel about permutation importance should help you to have a deeper understanding of which feature has a good predictive signal for a particular sample.
