I am trying to build an ensemble of three binary classifiers: A, B and C. Each one generates probabilities for the positive class. My goal is to generate a single probability for each case from the three probability estimates of each classifier. To simplify things, let's say that I want my ensemble probability for each case to be the mean probability estimated by each of the three classifiers. (My actual aggregation method will be more nuanced than that, but the simplification is sufficient for this question.)
My challenge is that each of the three classifiers is calibrated quite differently. So, classifier A might use probabilities of 0.8 to 1 for its highest estimates of the positive class; classifier B might use 0.4 to 1 for the positive class; classifier C might use 0.07 to 1 for the same positive class. Thus, taking the mean of the three uncalibrated sets of probability estimates would be essentially meaningless.
I think I could resolve this challenge by normalizing each classifier's probability estimates to be on the same scale. Here is my proposed strategy:
- Identify the optimal threshold for each set of probabilities based on the threshold that maximizes Yourden's J = sensitivity + specificity − 1. (See my further explanation below on why I choose Yourden's J as a target.)
- For each classifier's set of probabilities, scale estimates from 0 to the threshold to become 0 to 0.5 and scale estimates from the threshold to 1 to become 0.5 to 1.
- Now I can aggregate (average) the three sets of probability estimates with the meaningful interpretation that aggregate probabilities from 0.5 to 1 favour the positive class whereas those from 0 to 0.5 favour the negative class.
Please note that I am NOT trying to choose the best of the three classifiers. In my ensemble model, I will retain all three and I want to use the information from each. Some answers to seemingly similar questions about optimal thresholds have recommended using strictly proper scoring rules, but I do not see how these would help my situation:
I am not trying to choose the best of three models; I am trying to scale the three sets of probabilities for meaningful aggregation. I recognize and appreciate the importance of separating model evaluation from decision-making, but for me, the ultimate decision is not to accept my artificial threshold of 0.5. Once the ensemble model is completed, the analyst is free to then shift the threshold above or below 0.5 depending on the costs of errors. With this intention to shift threshold when it is time to make decisions, I do not see that my intermediate need for an artificial threshold compromises the quality of model building. Please clarify me if I am mistaken here; I would certainly like to learn on this point.
So, my questions are:
- Is my strategy generally sound for resolving my challenge of aggregating three different sets of probability estimates? Or is there a better way to achieve my goal (which might or might not involve strictly proper scoring rules)?
- If my strategy is sound, what is a suitable approach to scaling probability estimates such that they are normalized with the same optimal threshold? (The resulting optimal threshold does not need to be 0.5; it just needs to be identical across the three models.)