I have read that the posterior probabilities of Naive Bayes classifiers are unreliable. Is this true? and if so, in what sense, and why?

Specifically, I am interested to know if the probabilities can be used as some measure of confidence of the predictions. For example, can I say that higher posteriors are more likely to be correct, or use a threshold to say that if the posterior is below some value then it is low-confidence.

  • $\begingroup$ Do you have the source of that statement? $\endgroup$ – Robert Smith Sep 28 '13 at 16:55
  • 1
    $\begingroup$ When something is called "Naive Bayes", it usually means either "not really Bayes" or "Bayes with an obviously false assumption" such as independence. So your confidence becomes empirical: in machine learning you use training data, validation data and test data as part of this empirical process. $\endgroup$ – Henry Sep 28 '13 at 19:00
  • $\begingroup$ @RobertSmith for example here: scikit-learn.org/stable/modules/naive_bayes.html - "although naive Bayes is known as a decent classifier, it is known to be a bad estimator, so the probability outputs from predict_proba() are not to be taken too seriously". $\endgroup$ – Bitwise Sep 28 '13 at 23:36
  • $\begingroup$ @Henry empirically I found that the posterior probabilities are predictive of actual errors on the test set. The question is if this is merely a coincidence (I suspect not), or if I can expect this to also work on similar datasets. $\endgroup$ – Bitwise Sep 28 '13 at 23:39

Sorry, too long for a comment.

Very interesting. The quote

although naive Bayes is known as a decent classifier, it is known to be a bad >estimator, so the probability outputs from predict_proba() are not to be taken >too seriously

is taken from this paper that in turn is citing Bennett and Monti and Cooper. I don't think Bennett is very convincing since he only claims that "many people" think that NB produces estimates too close to 1 and 0, but gives no proof. He offers as additional evidence the fact that for long documents, there is a tendency to push values to 0 or 1, even when they are completely wrong. Monti and Cooper's paper is a bit tricky because they're comparing finite mixture models (FM), finite mixture augmented Naive-Bayes model (FAN) and Naive Bayes, so they tend to describe more about this comparison than to describe in detail why Naive Bayes are poor estimators.

On the other hand, you can find the same claim for decision trees and SVMs, so maybe people think that a given classifier gives poor estimates when it is not an issue intrinsically associated to that classifier.

I think we simply need better references and better evidence. I hope other people can provide more information.

| cite | improve this answer | |
  • $\begingroup$ Thanks, this is helpful. I would also be happy to hear what other people think. $\endgroup$ – Bitwise Sep 29 '13 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.