Handling NLP probability results

Question

I'm a programmer without a statistics background. I've been working with NLP lately to classify documents, and I'm pretty up to speed with NLP concepts. I've gotten to the point where the NLP software is returning classification probabilities for each category, and the probabilities are quite accurate.

The next task is how to use the result probabilities to assign category(s) to each unlabeled document. Specifically, my dataset has:

• Title
• Body

The categorization done for each is:

1. 30 categories, where each document must belong to one category, and at most two categories. If no category is a strong match, the document is assigned to an "Unknown" category.
2. 10 other categories, where each document is only associated with a category if there is a strong match, and each document can belong to as many categories as match.
3. 4 other categories, where each document must belong to only one category, and if there isn't a strong match the document is assigned to a default category.

I'm classifying the Title and Body separately. The Body produces the most accurate classifications, but the Title can help improve accuracy in some cases. Examples of result probabilities might be:

---

Document 1
----------
Title:
Category 1 0.950
Category 2 0.030
Category 3 0.020
Category 4 0.005
Category 5 0.005

Body:
Category 1 0.920
Category 2 0.050
Category 3 0.020
Category 4 0.020
Category 5 0.010

---

Document 2
----------
Title:
Category 1 0.572
Category 2 0.185
Category 3 0.092
Category 4 0.077
Category 5 0.074

Body:
Category 1 0.129
Category 2 0.785
Category 3 0.052
Category 4 0.032
Category 5 0.002

---

Document 3
----------
Title:
Category 1 0.455
Category 2 0.425
Category 3 0.102
Category 4 0.010
Category 5 0.008

Body:
Category 1 0.462
Category 2 0.449
Category 3 0.081
Category 4 0.006
Category 5 0.002

---

As a human, I can fairly easily analyze the results and see which category(s) should be assigned, whether no category is a strong match, etc.

The question is, how to do this in an automated way?

I almost feel like I should pass the results back through another prediction engine to see whether a particular category should be assigned or not. Or at run a large number of tests with different parameters to see which produces the most accurate results. Parameters for the weight of the title and description probabilities seem like obvious candidates. I have a very large training dataset and also a large testing dataset that was not used in the training.

Advice on the best way to handle this, and/or what is this particular problem called if it has a name?

Edit:

This problem seems to be called multi-label classification. There is a very comprehensive PDF on the topic. It contains a list of "thresholding strategies" that can be used. The authors have created an addon for Weka called Mulan that implements a number of multi-label learning and thresholding strategies.

At this point I need to either try Weka and Mulan, or implement one of the thresholding strategies using the results from my current NLP tool.

Answer 1

You don't describe what "NLP software" you're using, but your question seems to be around how to build a text-classification system. I build those very frequently and here are a few points that may help:

A multinomial classifier provides "scores", whether 0, 1 or something between on multiple classes per observation. It is possible to represent a multinomial classifier as multiple binomial classifiers, with each binomial classifier providing the score for each "category".

Sounds like you need 3 multinomial classifiers for each of your "buckets" of categories, each consisting of 31, 11 and 5 binomial classifiers, respectively.

From a practical/coding standpoint, this can be accomplished by supplying your text features (most likely some summary of word/sentence vectors) and binary encoded labels to a naive bayes or SVM model, with the binary encodings translating the category you want each model to predict to 1 and the other categories to 0.

You should be able to accomplish this with pandas, gensim (word2vec), and scikit.

If your question also encompasses how to expose your model to some broader application for the purpose of automated classifications, you can roll your own model management/deployment tools or you can look at something like Watson Machine Learning, that has a pre-built API and GUI tools for various tasks that are part of the model lifecycle.