# Unsupervised Bayesian naive Bayes

I'm reading a paper Gibbs sampling for the uninitiated.

In this paper, the authors try to use Gibbs sampling for a bayesian naive bayes model. They formalize the model as a graphical model in page 8. And in the example, they are trying to predict the emotion(sentiment) of a document.

However, what I don't understand is that, they claim without label $L$, using Gibbs sampling could still sample all the parameters needed, including $L$. I'm not sure how should I interpret this. Without training label, it's essentially a clustering problem, but if not using labels, how should we interpret the learnt label $L$?

• paper is here: cs.umd.edu/~hardisty/papers/gsfu.pdf – Eddie Xie Sep 15 '13 at 18:44
• Can you point out which page contains the interpretation you don't understand? – Robert Smith Sep 15 '13 at 20:17
• @RobertSmith Sure. So the part I don't understand is that, the authors claim without label they could still use gibbs sampling to estimate all the parameters, including $L$ (which should be the labels). However, I don't understand if so, how would they interpret the labels. In this paper they interpret it as "sentiment" because those are judged by human beings. But what if we don't ask human to label the data, and don't have a clue what the label should be? Shall we interpret it as "spam vs not spam" or "positive opinion/negative opinion"? That's confusion to me. – Eddie Xie Sep 17 '13 at 14:10
• I understand what you're asking, but I wanted to know which page of the paper contains the interpretation you're describing. – Robert Smith Sep 18 '13 at 6:29
• @RobertSmith Oh sorry about that. Page 7 section 2 "the features under consideration are the words in the document, and that the document-level class variable we’re trying to predict is a sentiment label whose value is either 0 or 1.". That's the "label" they are providing. – Eddie Xie Sep 18 '13 at 17:02

Too long for a comment.

In page 7 section 2, the authors clearly establish both labels "1" and "0" for the classes of their dataset. So, let's say "1" is "happy" and "0" is "sad". There you have your sentiment analysis.

Since they chose to use Naive Bayes as classifier, there are some parameters and hyperparameters to calculate in the Bayesian formulation. Such parameters are usually obtained integrating over all possible values (see 2.4.3). However, I think the point of this paper, is to show you that you can get away without calculating difficult integrals and instead, estimate conditional probabilities using Gibbs sampling (see 2.5.2).

At least, from what I have been able to look at, they're using labels to get an approximation of the joint distribution via Gibbs sampling.

• Thanks a lot Robert! Yes but they claim we could still estimate the labels if there's no ground truth labels. However, what should be the interpretations of such "learnt labels"? I tried to implement the paper yesterday. What I used 20-news data and labels are whether the doc is about sports vs science. I didn't give the model the lables, and use gibbs sampling to infer the labels and then compare the learnt labels with ground truth. It's accuracy is about 60%, close to random guess. So I'm guessing it can't really learn the concept given no labels at all (sports vs science in this case)? – Eddie Xie Sep 19 '13 at 17:47
• If you refer to this: Following Pedersen, we're going to describe the Gibbs sampler in a completely unsupervised setting where no labels at all are provided as training data., then the authors are stating they're going to describe Gibbs sampling as Pederson, probably for didactic reasons. But that doesn't mean that they're going to use it without labels. Actually, you can see after equation 49 they are sampling to obtain that conditional distribution and are distinguishing between label "1" and "0". Otherwise, you wouldn't know what parameters belong to each label. – Robert Smith Sep 19 '13 at 19:33

Probably the questioner has got the answer, its been a long time, but the accepted answer doesn't seem to answer the question well. The unsupervised naive Bayes text classifier tries to find the most prominant classes in the data set. Those may not be the classes you are looking for. You may be looking for a happy/sad classes but the most prominent classes may be.. say.. of present/past tenses, and the sampler would find tense class only.

In order to prevent the sampler from discovering unintended classes one need to seed the sampler with labeled data.Section 2.5.3 in the paper talks about just that, however the need for using labeled data is not mentioned. This allows the sampler to get some idea of the ground truth and then generalize better using unlabeled data.

The paper "Semi-Supervised Text Classification Using EM by Kamal Nigam et.al" shows how unsupervised learning can improve the results of supervised learning. Here the authors improve the results of supervised learning by adding to it unsupervised learning with unlabeled data. They say if that if the unlabeled data is much larger the performance decreases. This happens because the algorithm then drifts away to the most prominent classes in the data, not the ones you want.