Normalize different binary prediction probability thresholds

Question

I am trying to build an ensemble of three binary classifiers: A, B and C. Each one generates probabilities for the positive class. My goal is to generate a single probability for each case from the three probability estimates of each classifier. To simplify things, let's say that I want my ensemble probability for each case to be the mean probability estimated by each of the three classifiers. (My actual aggregation method will be more nuanced than that, but the simplification is sufficient for this question.)

My challenge is that each of the three classifiers is calibrated quite differently. So, classifier A might use probabilities of 0.8 to 1 for its highest estimates of the positive class; classifier B might use 0.4 to 1 for the positive class; classifier C might use 0.07 to 1 for the same positive class. Thus, taking the mean of the three uncalibrated sets of probability estimates would be essentially meaningless.

I think I could resolve this challenge by normalizing each classifier's probability estimates to be on the same scale. Here is my proposed strategy:

1. Identify the optimal threshold for each set of probabilities based on the threshold that maximizes Yourden's J = sensitivity + specificity − 1. (See my further explanation below on why I choose Yourden's J as a target.)
2. For each classifier's set of probabilities, scale estimates from 0 to the threshold to become 0 to 0.5 and scale estimates from the threshold to 1 to become 0.5 to 1.
3. Now I can aggregate (average) the three sets of probability estimates with the meaningful interpretation that aggregate probabilities from 0.5 to 1 favour the positive class whereas those from 0 to 0.5 favour the negative class.

Please note that I am NOT trying to choose the best of the three classifiers. In my ensemble model, I will retain all three and I want to use the information from each. Some answers to seemingly similar questions about optimal thresholds have recommended using strictly proper scoring rules, but I do not see how these would help my situation:

• Normalise different thresholds for binary prediction
• Reduce Classification Probability Threshold

I am not trying to choose the best of three models; I am trying to scale the three sets of probabilities for meaningful aggregation. I recognize and appreciate the importance of separating model evaluation from decision-making, but for me, the ultimate decision is not to accept my artificial threshold of 0.5. Once the ensemble model is completed, the analyst is free to then shift the threshold above or below 0.5 depending on the costs of errors. With this intention to shift threshold when it is time to make decisions, I do not see that my intermediate need for an artificial threshold compromises the quality of model building. Please clarify me if I am mistaken here; I would certainly like to learn on this point.

So, my questions are:

• Is my strategy generally sound for resolving my challenge of aggregating three different sets of probability estimates? Or is there a better way to achieve my goal (which might or might not involve strictly proper scoring rules)?
• If my strategy is sound, what is a suitable approach to scaling probability estimates such that they are normalized with the same optimal threshold? (The resulting optimal threshold does not need to be 0.5; it just needs to be identical across the three models.)

Answer 1

Since you have seen previous threads on thresholds, you may already have seen this thread on accuracy: Why is accuracy not the best measure for assessing classification models? Note that the exact same problems also apply to sensitivity and specificity. So I am already skeptical about using these inherently flawed measures in your step 1.

Your component classifiers are not just calibrated differently, they are miscalibrated. If one classifier outputs a predicted probability of 0.8 for an instance to be of the target class, and the next one gives 0.6, and the third one gives 0.2, then something is off, because at least two of them have a very wrong idea of the true probability - no matter whether these numbers exceed the thresholds. What these classifiers give may be some KPI that is correlated with the true probability of target class membership, but it is quite obviously not a good estimate of it.

Thus, I would say the first order of business would be to recalibrate your classifiers. Happily, this is quite simple. For each classifier, run a logistic regression for the target class on the classifier's predicted probabilities. (Use splines if necessary to account for nonlinearities.) The fitted outputs from these logistic regressions should be far better calibrated. I recommend you take a look at proper scoring rules for these recalibrated classifiers (and the miscalibrated ones, too), just to get an idea of how they perform.

Once you have recalibrated classifiers, you can simply use the arithmetical average of the recalibrated predicted probabilities. And then you or your user can apply one or multiple thresholds based on costs as they see fit.