2
$\begingroup$

I am using VotingClassifier from sklearn.ensemble however i am puzzled with the results. Consider following algorithms:

alg1 = LogisticRegression()
alg2 = GradientBoostingClassifier()
alg3 = GaussianNB()

Then when I ensemble and fit the models with following weights:

eAlg = VotingClassifier(estimators=[('LR',alg1),('GB',alg2),('NB',alg3)],
                        voting='hard',
                        weights=[1,5,1])
eAlg.fit(X_train,y_train)
pred_ens = eAlg.predict(X_test)

I get following result:

np.mean(pred_ens == y_test) #0.91

(I am predicting only 0/1 classes)

Just for my curiosity, I put weight=5 to GB. Doesn't it mean that even if other two classifiers says otherwise, the model will follow GB? So it should be equal to GB alone.

Because if i train only GB alone, i get slightly higher score. How is that possible?

$\endgroup$
  • $\begingroup$ very strange, can you get the individual predictions and find out which records it is differing for? $\endgroup$ – Max Flander Jan 14 '16 at 3:47
  • $\begingroup$ Yes, i can. But that doesn't say WHY. $\endgroup$ – HonzaB Jan 14 '16 at 11:13
  • 1
    $\begingroup$ dont get angry, i'm trying to help, why don't you take a look a the code, use pdb.set_trace() see if you can isolate where the problem is occurring. if you post a minimal dataset which results in this behaviour, i can take a look as well. $\endgroup$ – Max Flander Jan 14 '16 at 21:52
2
$\begingroup$

You set the voting parameter to hard. Afaik that means your weights array will be ignored, as it is only used with soft voting. You get lower results than with the single classifier, because the other two learned models win a vote against the "better" classifier for too many of the data points, where they are actually wrong. At least that is my guess.

$\endgroup$
  • $\begingroup$ You are wrong about the weights being used only in soft voting. But, overall, your post is correct - it seems that the reason for the observed behaviour is that the "wrong" classifiers over-power the correct one. $\endgroup$ – Nick Jan 14 at 20:55
1
$\begingroup$

I tried the something similar and got the expected results. In your case, this must be due to not setting the parameter 'random_state'. If you do not set a SEED or 'random_state' to the classifier definition then it results in different accuracies each time you run.Try,

alg1 = LogisticRegression(random_state=20) alg2 = GradientBoostingClassifier(random_state=20) alg3 = GaussianNB()

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.