I start to work with ensembe methods these days focusing on stacking. I am wondering whether to us each models class probability ( real number in $[0,1]$) or the classifcation itself (in the binary case an integer in $\{0,1\}$) as input.

In my feeling the probabilites are more natural as it is a difference if I classify as $0$ because the probability is $0.1$ or because it is $0.49$ and I lose exactly this information if I apply the ensemble algorithm directly on the class.

What is the usual approach? Is there a reference on how to construct ensemble classifiers using stacking?

PS: Ensembling a gradient boosted model and a logit-model I got slightly better results using the classification than the probabilities - but I think this was the case because the probabilities accumulated mostly around $0.5$ and thus the difference did not matter that much.

up vote 6 down vote accepted
+50

What you're asking is the difference between voting (using classifications) and averaging (using probabilities), and as far as things go, it's up to preference and/or performance. Try both and see what works best, and if there is not a significant difference, pick one that you feel better about.

I'll try to explain the feeling better part now.

Voting might sound a bit un-natural as you've said since it ignores if the probability is 0.1 or 0.49 - however take this example.

Gradient boosted tree (GBT) models are really prone to overfitting. That's why all the parameters need to be tweaked carefully and why a small change leads to a massively better generalized performance (check kaggle for reference).

In this case, overfitting might mean that the GBT model will be very sure for an instance with $p_1 = 0.85$. However, two different, simpler models will not be that sure and will output $p_2 = 0.39$ and $p_3 = 0.32$ respectively.

The average is $p_{avg}=0.52$ and the sample is classified according to the GBT model, even though 2/3 of your models did not agree with this. In case that you used voting, the classification would obviously be different.

So, what's the conclusion here? If some of your classifiers are prone to overfitting hard and providing large probability errors for missclassified samples (which you can test for with various loss measures) but performs well otherwise, perhaps you should use voting for the error to be fixed by other classifiers.

If the probability errors for the missclassified samples are within reasonable bounds, then you might be better off using probabilities, since there is no information loss and that is usually a good thing.

Overall, I doubt there is an absolutely correct way to do this, and it definitely differs from task to task and ensemble to ensemble.

  • Thank you for your explanation, this is the input that I was looking for. – Richard Dec 4 '15 at 12:20

The answer by mttk is quite nice. I will just add that you could also extract the probabilities and use them as input to a meta-classifier such as a simple logistic regression. This will automatically adjust for differences in the way classifiers produce probability estimates (i.e. classifier A almost exclusively produces probabilities on the range [0.4, 0.6], whereas classifier B almost exclusively produces probabilities near 0 or 1).

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.