Distinguishing objective from subjective text using a Naive Bayes classifier

Question

I am trying to built a classifier for subjective and objective text using imdb data. For objective data point I am using the movie's plot summary as input. For subjective data points I am using reviews of the movie. For example

Objective (plot of Abandon)

Catherine Burke is under pressure. She faces exams, completion of her thesis, and a competitive interview process, all of which is compounded when a police detective, Wade Handler, begins investigating the two year-old disappearance of her boyfriend, Embry Langan, a young man whose memory haunts and obsesses her. As the investigation continues, Catherine is forced to choose between her past passions and new possibilities, even as Handler is discovering surprising new facts about Embry and his possible connection to another disappearance from campus.

Subjective (One of the user reviews)

Caught this on cable last night and I liked it. I thought Katie Holmes did extremely well with a very tricky role, and I thought there were a lot of well written exchanges between the characters, excellent atmospheric touches, and enough psychological ambiguity to allow me to figure out what was really going on before the ending, but this didn't make the film predictable - it made it clever. And the title is a good one - extremely telling, a clue in itself. Of course, it's not a perfect film by any stretch; there's too much stuff that really doesn't need to be in the movie but I still give it a 6 (my IMDb equivalent of *** - a decent premise, decently executed).

I took complete plot summary as one data point where as in case of reviews each review by a single user is a single data point. In my database different reviews of the same movie by different users are entered as different data points.

After this I cleaned the words of special character, removed stop words, calculated the Information gain and applied Naive Bayes to build the classifier.

My questions are:

1. Is my algorithm to build the classifier correct?
2. My classifer is heavily biased toward classifying text as objective. Am I making mistake in the creation of my training data?
3. I want to create a generic classifer that can be used for tweets or
   text extracted from blogs. Is movie review data sufficient? Right now it's not working even for movie review data.

Answer 1

Regarding class imbalance in training data, for estimating the model parameters - the more data the better. Depending on the software implementation that you are using (I assume you are using), generally, the model would be optimized for the same distribution of classes as it is in the training data. It you expect your twitter or other data to have a very different class balance, then you may want to adjust your predictions. Assuming equal costs of errors, you can do it as follows.

A Naive Bayes implementation would typically output $p(sub|X)$ and $p(obj|X)$, where $obj$ and $sub$ are the classes, and $X$ is a vector of word frequencies. You classify an observation to the class that has a higher probability. A posterior probability can be decomposed into $p(sub|X) = p(sub)p(X|sub)/p(X)$, this is how Naive Bayes does it. However, $p(sub)$ is estimated on the training data. We need to replace it by $p^\star(sub)$ which is the expected probability of subjective texts in the application data. For example, if you expect the amounts of objective and subjective tweets to be equal, you can set $p^\star(sub) = 0.5$.

Now the adjustment. Rearranging the terms of the previous equation we get $p(X|sub)/p(X) = p(sub|X)/p(sub)$. The adjusted posterior probability for application is $p^\star(sub|X) = p^\star(sub)p(X|sub)/p(X)$. We plug in the rearranged equation and get $p^\star(sub|X) = p(sub|X)p^\star(sub)/p(sub)$. We do the same for $p^\star(obj|X)$. Then an observation is classified to a class that has a higher probability $p^\star(class|X)$, instead of $p(class|X)$.

I hope this helps.