Naive Bayes classification for "That's what she said" problem

Question

I became interested in doing this in C# for my own amusement after reading the following papers:

http://www.cs.washington.edu/homes/brun/pubs/pubs/Kiddon11.pdf

I also took a look at http://www.cs.rpi.edu/academics/courses/fall03/ai/misc/naive-example.pdf as a concrete example for my implementation.

I have a working implementation now, but I wanted to make sure that I was approaching it properly. I just want to have a solid Naive Bayes Classifier (unigram).

Problem statement and setup

I am using two sets of data, a list of sentences that ARE "that's what she said" and a list of sentences that dont make sense with a "that's what she said" suffix.

Next I parse through all the words in all of the sentences and keep a tally on each word and how many times it was found in each of the two sets, so I might end up with data that looks like this:

Word         PositiveCount             NegativeCount
wet          23                        4
hard         30                        5
haiti        0                         20
to           60                        77

The I iterate over all of the words and calculate individual $\Pr(\text{Positive}|\text{<word>})$ and $\Pr(\text{Negative}|\text{<word>})$ using the following formula which I found in the above example paper:

P(Positive|wet) = (23 + p * m) / ((23 + 4) + m)
P(Negative|wet) = (4 + p * m) / ((23 + 4) + m)

Where m is the equivilent sample size and p is the a priori estimate. Then in order to check an unknown sentence to see if it is a TWSS I iterate over each word in the sentence and multiply their positive [probibility distributions?] together and multiply all that by p. And do the same for the negative. Then if the positive number is larger I say that the sentence is a "that's what she said".

Questions

1. Currently I am using $p = .5$ for both positive and negative. I feel like I could be doing something better. Is this what the Bayesian vs. frequentist thing is about? How would I go about getting better numbers for $p$?

2. Also, I am using $m$-estimates for the $\Pr( \text{Yes/No} | \text{<word>} )$. Should I be doing it this way and what should $m$ be? What effects does it have to make $m$ larger/smaller?

Super Minor Question: Suggestions on where to get sample data from would be a bonus.

Answer 1

1. You are not using Naive Bayes, you are actually using something I'd call "Multiplicative Decision Stump"-Classifer ;). You can do that, but I'd recommend in this case to calculate the micro or macro-average across all words in the sentence (instead of multiplying them). E.g. macro-average: $p(Positive|sentence)=\frac{1}{n}\sum_{word\in sentence}p(Positive|word)$
2. I'd set $p$ to $\frac{Positive}{Positive+Negative}$ for calculating the $p(Positive|.)$ and $\frac{Negative}{Positive+Negative}$ for calculating the $p(Negative|.)$ respectively
3. $m$ is the weight of the prior meanwhile the word-count is the weight of the occuring word. The lower $m$ the more importance the probability calculated by word-frequency gets and vice versa. Let's say for example $m$=8 and word-count=2 (for a certain word), than the resulting score will contain of 80% prior-information and 20% non-prior-information. Hence I'd always compare $m$ to the word-count of important words (i.e. those whose yes/no-probability differs strongly from the prior). Unfortunately, there is no "golden hammer"-value for this variable, so I suggest to play around a little bit.
4. If you want to try out Naive Bayes, I suggest the section "Documentation classification" in the wiki-article about Naive Bayes

Answer 2

Here is a Web site that uses a classifier to determine the gender of an author of text:

       http://bookblog.net/gender/genie.php

There are a number of articles on the subject at the Web site and the composer of the site has written a number of articles on the subject as well. There are a number of good methods that can be used in the classification

1. Bayesian Logistic Regression
2. Linear Discriminant Analysis
3. Quadratic Discriminant Analysis
4. Bayes Classifiers

Have fun

Answer 3

An interesting source of suggestions for this problem might be the That's What She Said Quora thread. Specifically, the first comment identifies that it might make sense to use Twitter streams as a source of statements to which there has been a response of "That's What She Said". Another thing you could do would be to make a simple UI which would take input from twitter, and then ask a human annotator to decide whether "That's What She Said" is an appropriate response.

The UW paper mentions their data sources; I assume (though you didn't mention it specifically) that you're already using this data? (It looks like you can slightly increase the size of the training set from that paper, but not a lot.)

Another potential source of data to look at might be to analyze television show dialogue for positive examples. (The "A Computer That Knows When To Say “That’s What She Said" Forbes Article mentions The Office, for example.) Using Hulu's Captions search (search for That's What she said) identifies 179 examples, though it looks like in the first 10, only 2 are positive training examples, so that may be a somewhat noisy source of data. Officequotes.net appears to have a larger source of potential data, though again, cleaning it up and using it may take some work.