2
$\begingroup$

This question had been asked several times in here, but I think I have something new to add.

I'm interested in predicting if some specific event will happen (binary classification). I have two distinct methods that generate a prediction about it. The first methods, an XGBoost model, gives a binary response and the probability of the event (as the probability of being in that class). The second method is an empirical model that has as output a binary classification, but with I can estimate the probability of event given the classification.

Now, the question is, how to combine the two?

My proposal, and here is where I think I didn´t saw anything similar, is the following:

if both classifiers agree on prediction, then that's the prediciton. When they disagree, I want to compute the weighted probability of event, where I will give more weights to extreme probabilities. The logic is that there is more information in a classifier that tells you there is a 90% or a 1% change of event compared to one that says 50%.

To do that I compute the weight $w_i$ as $|\ln\frac{p_i}{1-p_i}|$, with $p_i$ as the probability of event. What do you think

Improve this question
$\endgroup$
1
  • $\begingroup$ I think thats an interesting question, especially because the prediction itself is binary. If i understand your approach right, it would mean, that each time model1 gets a weight of > 1 it takes the "win" and if it gets a weight < 1 model 2 takes the win. If im right the breakingpoint towards model 1 should be between probability 0.73 and 0.74. Is this what you want to archive? $\endgroup$
    – TinglTanglBob
    Commented Mar 13, 2019 at 10:23

1 Answer 1

Reset to default
2
$\begingroup$

there is more information in a classifier that tells you there is a 90% or a 1% change of event compared to one that says 50%

Why do you think so?

If one classifier outputs more extreme probabilities, that first of all indicates that it is more confident. This certainly does not indicate that it is better. I can easily build you an extremely confident classifier that is extremely bad! (Simply output "probabilites" of 0% or 100% at random.)

The goal of any probabilistic classifier should be good calibration: when it outputs a 90% chance that a case belongs to class A, it should in fact be A in 90% of cases. If it is A in 85% or in 95% of cases, then the classifier is just as poorly calibrated as if it output 50% probability but was correct only in 45% or 55% of cases. Take a look at proper to assess calibration.

That said, of course your particular problem and classifier could be improved by such a scheme. That is an empirical question. I would just be careful about claiming that higher confidence implies higher competence. That simply doesn't work. Neither for classifiers nor for humans.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the input. Indeed, more extreme probabilities just reflect degree of confidence, which might not be correlated with competence (for both, yes :) ). However, my point was, if I trust both my classifiers equally, does it make sense to give more weight to the more confident one? $\endgroup$
    – Diogo Santos
    Commented Mar 13, 2019 at 16:42
  • $\begingroup$ It does make sense if your more confident classifier is actually better, as per my last paragraph. I don't see a principled reason why your approach should work or not. Just try it with a large enough holdout set, against simple alternatives like unweighted combinations (which are often better than estimating "optimal" weights, which has been called the "forecast combination puzzle"). $\endgroup$
    – Stephan Kolassa
    Commented Mar 13, 2019 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.