Text mining discrete paragraphs

Question

I have a spreadsheet with Titles in the first column, descriptions in the second column and whether or not they were included in our screening in the third column. We are attempting to find out which individual words were related to inclusion versus exclusion so that we can use them in a much larger data set. From my brief understanding of text mining, I think this would require a Naive Bayes or Logistic Regression method. Is this correct or is there a more appropriate method?

---

Preferably, without knowing the words in advance, I would hope to identify which are related to inclusion vs. exclusion.

An example:

Title: Excel 2010 Advanced Description: This free Excel 2010 eBook should be used as a point of reference after following attendance of the advanced level Excel 2010 training course. It covers all the topics taught and aims to act as a support aid for any tasks carried out by the user after the course.

Title: Excel 2003 Formulas Description: Everything you need to know about Mastering operators, error values, naming techniques, and absolute versus relative references; Debugging formulas and using the auditing tools; Importing and exporting XML files and mapping the data to specific cells; Using Excel 2003's rights management feature; Working magic with array formulas; Developing custom formulas to produce the results you need

So if these two were considered to be included then for example the word "Excel" would be a common predictor.

I am sure that my question might be naive but am trying to understand the scope of the project and whether or not I can handle it, and how much assistance I will require to accomplish my goals.

Thanks in advance for all your help and insight.

AMAS

Answer 1

Yes, a Naive Bayes model would be a good start to approaching this problem. A classic use of Naive Bayes is to classify emails as spam or not spam. http://en.wikipedia.org/wiki/Bayesian_spam_filtering

Your problem is similar but the categories are inclusion and exclusion. So you would begin by creating a numerical representation for each description field. A common way to do this is to create a 'bag' of the most common words in all your descriptions. Then for each description field check to see if it contains the words in the 'bag'. Create a vector that has an entry for each word in the bag, 1 if the description contains the word and 0 if it doesn't. Then use these vectors to train a Naive Bayes classifier. Then you can use it to classify descriptions in your larger data set.

If your description fields have considerably variable length you may want to try a tf-idf representation instead of binary. You might also try using Logistic Regression or Support Vector Machine classifiers as well to see what works best. Many languages have packages for these classifiers. For example in python you can use the sklearn package.

Answer 2

I take the information theoretic approach (which also requires a classifier--maybe Naïve Bayes, maybe SVM). To do this, I usually flip the question on its head: what features tended to be most informative for determining class. The wikipedia entry is a pretty decent reference on the specifics of mutual information. It's pretty straightforward to implement this method in a scripting language, such as Python!