I have created $n$ classifiers for binary classification. I am given some data to assign some probabilities to a sample that belongs to either class 0 or class 1.

What I want to do is to combine the probabilities (that a sample belongs to class 1) that I get from the $n$ different classifiers, however I want to assign some coefficients to the prediction of each classifier, since not all the classifiers have the same predictive power.

My question is how to choose the weight to apply to the calculated probability of each classifier.


There are two methods in my mind:

(1) The manual way. Since you have $n$ classifiers and their predictions, if you did train-test-split to test the accuracy, you will have a sense of which classifier(s) are doing better and which are doing worse. You can use this accuracy as a weight assignment for your ensemble: more weights to better classifiers, less weights to worse performing classifiers.

(2) Stacking. You can create another classifier (I usually choose Logistic Regression) with input being the predictions from your $n$ classifiers, and label being the true label for your training data. This way, this new classifier will determine the weight automatically. It usually gives you a better performance than models without ensembles. You can read more about it here: http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.