I've built a logistic regression classifier that is very accurate on my data. Now I want to understand better why it is working so well. Specifically, I'd like to rank which features are making the biggest contribution (which features are most important) and, ideally, quantify how much each feature is contributing to the accuracy of the overall model (or something in this vein). How do I do this?
My first thought was to rank them based on their coefficient, but I suspect this can't be right. If I have two features that are equally useful, but the spread of the first is ten times as large as the second, then I'd expect the first to receive a lower coefficient than the second. Is there a more reasonable way to evaluate feature importance?
Note that I'm not trying to understand how much a small change in the feature affects the probability of the outcome. Rather, I'm trying to understand how valuable each feature is, in terms of making the classifier accurate. Also, my goal is not so much to perform feature selection or construct a model with fewer features, but to try to provide some "explainability" for the learned model, so the classifier isn't just an opaque black-box.