# Ensembling with VotingClassifier

I am using VotingClassifier from sklearn.ensemble however i am puzzled with the results. Consider following algorithms:

alg1 = LogisticRegression()
alg3 = GaussianNB()


Then when I ensemble and fit the models with following weights:

eAlg = VotingClassifier(estimators=[('LR',alg1),('GB',alg2),('NB',alg3)],
voting='hard',
weights=[1,5,1])
eAlg.fit(X_train,y_train)
pred_ens = eAlg.predict(X_test)


I get following result:

np.mean(pred_ens == y_test) #0.91


(I am predicting only 0/1 classes)

Just for my curiosity, I put weight=5 to GB. Doesn't it mean that even if other two classifiers says otherwise, the model will follow GB? So it should be equal to GB alone.

Because if i train only GB alone, i get slightly higher score. How is that possible?

• very strange, can you get the individual predictions and find out which records it is differing for? – Max Flander Jan 14 '16 at 3:47
• Yes, i can. But that doesn't say WHY. – HonzaB Jan 14 '16 at 11:13
• dont get angry, i'm trying to help, why don't you take a look a the code, use pdb.set_trace() see if you can isolate where the problem is occurring. if you post a minimal dataset which results in this behaviour, i can take a look as well. – Max Flander Jan 14 '16 at 21:52

You set the voting parameter to hard. Afaik that means your weights array will be ignored, as it is only used with soft voting. You get lower results than with the single classifier, because the other two learned models win a vote against the "better" classifier for too many of the data points, where they are actually wrong. At least that is my guess.
alg1 = LogisticRegression(random_state=20) alg2 = GradientBoostingClassifier(random_state=20) alg3 = GaussianNB()