Why does nobody use the Bayesian multinomial Naive Bayes classifier?

Question

So in (unsupervised) text modeling, Latent Dirichlet Allocation (LDA) is a Bayesian version of Probabilistic Latent Semantic Analysis (PLSA). Essentially, LDA = PLSA + Dirichlet prior over its parameters. My understanding is that LDA is now the reference algorithm and is implemented in various packages, while PLSA should not be used anymore.

But in (supervised) text categorization, we could do exactly the same thing for the multinomial Naive Bayes classifier and put a Dirichlet prior over the parameters. But I don't think I have ever seen anyone do that, and the "point estimate" version of multinomial Naive Bayes seems to be the version implemented in most packages. Is there any reason for that?

Answer 1

Here is a nice paper that addresses some of the 'systemic' shortcomings of the Multinomial Naive Bayes (MNB) classifier. The idea is that you can boost the performance of MNB through some tweaks. And they do mention using (uniform) Dirichlet priors.

Overall if you're interested in MNB and you haven't read this paper yet, I would strongly recommend to do so.

I also found an accompanying MSc thesis by the same person / people but haven't read it myself yet. You can check it out.

Answer 2

I suspect most NB implementations allow for the estimation of the conditional probabilities with the Laplace correction, which gives a MAP solution to the Bayesian NB classifier (with a particular Dirichlet prior). As @Zhubarb (+1) points out, Bayesian treatments of NB classifiers have already been derived and implemented (Rennie's thesis/papers are well worth reading). However, the independence assumption of NB is almost always wrong, in which case making the model more strongly dependent on that assumption (via a full Bayesian treatment) might not be a good thing to do.

Answer 3

I do not believe what you describe is true. The probabilistic models for LDA and MNB are different.

One main difference between the two is that in the generative model for LDA, when a word is drawn, first a topic for that word is chosen, and then a word from that topic distribution is chosen. I.o.w. each word in a document can be drawn from a different topic.

In the generative model for MNB, the document is assigned one class and all the words in that document are drawn from the (same) distribution for that class.