Binning Calibrated probability scores for business use Context:
We have a model that outputs calibrated probability scores for a binary classification problem (events/nonevents). There is a general business requirement that we bin these outputs further to make the predictions easier for business usage. For example, it is encouraged that we overlay the probability scores with scores of 1 to 5, where 5 will represent prob. scores of say 10%-100%, score 4 will do the same for scores of say 5%-10%...etc. And say one arrives at the bin edge 10-100% because the area contains 90% of the minority class (events) in the test sample and also is 2% of the entire population, and similarly one arrives at bin edge 5-10% because the area contains an additional 5% of the minority class (events) in the test sample which is also 25% of the total population, so by reviewing bins 4 and 5 (5% to 100%  prob. scores) one reviews only 27% of the population but captures 95% of the events.
The question is:
This binning process is extremely qualitative and subject to criticism (why 5 bins, why should we believe 90% of the event population will remain in the 10%-100% in production setting...), does anyone know of a robust method to post-process calibrated probabilities into bins for decision making, without breaking this fundamental rule of machine learning "In general, with a multistep modeling procedure, cross-validation must be applied to the entire sequence of modeling steps" ESL II."?
Any resources and links would be appreciated.
 A: If you need to bin the predictions for business reasons, you probably want to do something like treating the bins differently, e.g. assigning them to different marketing campaigns. In such a case, you have in mind optimizing some objective (money, clicks, subscriptions, churn, etc). What you need to do is find such bins that are optimal given the objective. This is an optimization problem. Any other binning strategy would be arbitrary.
A: To follow up on what @Tim said, Let me give some specific examples.
Optimizing for marginal revenue of investment: cut at the first edge that accounts for X% of revenue increase. X is decided by the amount required to increase head count in our department in next quarter, cause HR has asked for support evidence. We will make a case to hire new people to support customers in this first bin with the expected revenue it will bring.
Optimizing for CTR: CTR is usually so skewedly distributed that a small percentile accounts for most data points. So a summary of the large group can inform decision on most of the customers. Here we are trading off complexity of the summary and the analysis coverage of the customers. If you wanna cover more customers in your summary, your conclusion and recommendation is harder to communicate to the business sector. Why 95% but not 98%? Cause the extra 3% looks different enough to complicate the conclusion without adding much to the recommendations.
