1
$\begingroup$

Context:

We have a model that outputs calibrated probability scores for a binary classification problem (events/nonevents). There is a general business requirement that we bin these outputs further to make the predictions easier for business usage. For example, it is encouraged that we overlay the probability scores with scores of 1 to 5, where 5 will represent prob. scores of say 10%-100%, score 4 will do the same for scores of say 5%-10%...etc. And say one arrives at the bin edge 10-100% because the area contains 90% of the minority class (events) in the test sample and also is 2% of the entire population, and similarly one arrives at bin edge 5-10% because the area contains an additional 5% of the minority class (events) in the test sample which is also 25% of the total population, so by reviewing bins 4 and 5 (5% to 100% prob. scores) one reviews only 27% of the population but captures 95% of the events.

The question is:

This binning process is extremely qualitative and subject to criticism (why 5 bins, why should we believe 90% of the event population will remain in the 10%-100% in production setting...), does anyone know of a robust method to post-process calibrated probabilities into bins for decision making, without breaking this fundamental rule of machine learning "In general, with a multistep modeling procedure, cross-validation must be applied to the entire sequence of modeling steps" ESL II."?

Any resources and links would be appreciated.

$\endgroup$
1
  • 1
    $\begingroup$ Yeah I have dealt with this before as well. Often I do it via constraints, e.g. business wants to prioritize N events in a queue, e.g. top 1000 are bin 1, next 1000 are bin 2, etc. (To test calibration later on make sure to cache the original probability prediction.) $\endgroup$
    – Andy W
    Nov 27 '21 at 19:02
3
$\begingroup$

If you need to bin the predictions for business reasons, you probably want to do something like treating the bins differently, e.g. assigning them to different marketing campaigns. In such a case, you have in mind optimizing some objective (money, clicks, subscriptions, churn, etc). What you need to do is find such bins that are optimal given the objective. This is an optimization problem. Any other binning strategy would be arbitrary.

$\endgroup$
3
  • $\begingroup$ I hear you. But I can't ignore the fact that we would be optimizing on one dataset and proceeding to use the results (bin thresholds) on outside samples. You can't really use the outcomes that have been created from a generating process that wasn't designed for it, that is why we do cross validation for model development. Cross validation ensures that the model is tuned such that it will work on out sample data, optimization on a sample data does not ensure that. I wonder if one can do the type of optimization you are mentioning with a cross validations mechanism. $\endgroup$
    – PaulNoah
    Nov 28 '21 at 22:34
  • 1
    $\begingroup$ @PaulNoah you give us almost any details so it is impossible to give detailed, problem-specific, answer. How this could be done and validated depends on the details of your specific problem. If you use some arbitrary binning you’re also not doing anything to validate it or make it optimal, not even locally. $\endgroup$
    – Tim
    Nov 28 '21 at 22:43
  • $\begingroup$ do you have any other thoughts on the subject that does not require optimization? Broad ideas will be fine. At this point I will be very happy camper if you got anything else, even if they are just links. Optimization is not a feasible option for the type of problem I am trying to solve. $\endgroup$
    – PaulNoah
    Nov 28 '21 at 23:03
0
$\begingroup$

To follow up on what @Tim said, Let me give some specific examples.

Optimizing for marginal revenue of investment: cut at the first edge that accounts for X% of revenue increase. X is decided by the amount required to increase head count in our department in next quarter, cause HR has asked for support evidence. We will make a case to hire new people to support customers in this first bin with the expected revenue it will bring.

Optimizing for CTR: CTR is usually so skewedly distributed that a small percentile accounts for most data points. So a summary of the large group can inform decision on most of the customers. Here we are trading off complexity of the summary and the analysis coverage of the customers. If you wanna cover more customers in your summary, your conclusion and recommendation is harder to communicate to the business sector. Why 95% but not 98%? Cause the extra 3% looks different enough to complicate the conclusion without adding much to the recommendations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.