I have input X, with 22 binary features and 70000 examples. The target y is one of 4 possible categories. They are unbalanced, with the most common having a bit more than 51% of the data. When I train a naive bayes classifier, it's cross validation accuracy is 47%. I initially thought it was overfitting, but the train score is also around 47%. What could be happening?

from sklearn import naive_bayes
from sklearn.model_selection import cross_val_score

#Naive Bayes
for alpha in [0,1,2,4,8,16]:
    bnb = naive_bayes.BernoulliNB(alpha=alpha)
    print alpha, cross_val_score(bnb, X_dumb[:train_size,:], y_train, scoring='accuracy')
  • $\begingroup$ It's probably just the fixed threshold of 0.5 used by the scoring. You might want to look into auc measure instead $\endgroup$ – seanv507 Jan 27 '17 at 7:26
  • $\begingroup$ AUC is not applicable - I have 4 classes, not 2. $\endgroup$ – Hristo Buyukliev Jan 27 '17 at 11:42
  • $\begingroup$ i see, well you might well look into the issue I mentioned. which generalises to choosing the max class probability (i guess). Why don't you use multinomial regression $\endgroup$ – seanv507 Jan 27 '17 at 13:21
  • $\begingroup$ I could, and it at least overperforms the benchmark of always predicting the most common category, but I want to compare several algorithms. Naive bayes should not perform worse than always predicting the most common entry. $\endgroup$ – Hristo Buyukliev Jan 27 '17 at 20:16
  • $\begingroup$ It's not a question of Naive bayes' it's a question of the decision rule you use, based on the raw probability you get from naive Bayes. Since nb doesnt give true probability (except for independent features), you can't expect nb to give you the best answer using max probability class, but you can change the rule (eg max where most popular class is x%lower than max 'probability' output class). Make sure you understand binary problem before multiclass $\endgroup$ – seanv507 Jan 28 '17 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.