I'm running some ensembles of several algorithms and I'm trying to establish possible rules for the final classification process.

Right now if have the class probabilities from 3 models. Its a binary classification so, in each prediction i have two columns

  • Class_1
  • Class_2

And each row has the probability for both classes.

Suppose Class_2 is the 'positive' outcome. I have a cutoff value where if the probability of Class_2 is greater than said value, I classify the row as Class_2, otherwise its Class_1. That is what I do with each model.

Now, since I have three models (I plan to extend it to 5 in the future), I've been trying the following rules, to build a 'blended' prediction.

  • Average probability in each model, and check against the cutoff value
  • Voting, return the cutoff value if a majority of models have a probability greater or equal to it
  • Maximum probability of Class_2 per row, basically i return the greatest of the three probabilities of Class_2 and compare it againts the cutoff value.

Now, I'm not really sure if the third rule is right. Is it? On the other hand I'd like to know what other rules can I implement?

  • $\begingroup$ I don't think the third rule is right. Suppose you have one bad model which is constantly over predicting the probabilities (and predicting higher probabilities than the other models); extreme case is a model that always gives a probability of 1. This rule then only uses the classifications of this bad model. Of course you can't beforehand know which model will be over or under predicting. But therefore you combine the classifications of the models and not select one model $\endgroup$ – Marcel10 Nov 16 '16 at 9:38
  • $\begingroup$ Yeah, I thought as much. But i gave it a try nonetheless. With my current data I didn't suffer from this problem, but i'll remove it from my tests. $\endgroup$ – Carlos Grappa Nov 18 '16 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.