# Handling NLP probability results

I'm a programmer without a statistics background. I've been working with NLP lately to classify documents, and I'm pretty up to speed with NLP concepts. I've gotten to the point where the NLP software is returning classification probabilities for each category, and the probabilities are quite accurate.

The next task is how to use the result probabilities to assign category(s) to each unlabeled document. Specifically, my dataset has:

• Title
• Body

The categorization done for each is:

1. 30 categories, where each document must belong to one category, and at most two categories. If no category is a strong match, the document is assigned to an "Unknown" category.
2. 10 other categories, where each document is only associated with a category if there is a strong match, and each document can belong to as many categories as match.
3. 4 other categories, where each document must belong to only one category, and if there isn't a strong match the document is assigned to a default category.

I'm classifying the Title and Body separately. The Body produces the most accurate classifications, but the Title can help improve accuracy in some cases. Examples of result probabilities might be:

Document 1
----------
Title:
Category 1 0.950
Category 2 0.030
Category 3 0.020
Category 4 0.005
Category 5 0.005

Body:
Category 1 0.920
Category 2 0.050
Category 3 0.020
Category 4 0.020
Category 5 0.010


Document 2
----------
Title:
Category 1 0.572
Category 2 0.185
Category 3 0.092
Category 4 0.077
Category 5 0.074

Body:
Category 1 0.129
Category 2 0.785
Category 3 0.052
Category 4 0.032
Category 5 0.002


Document 3
----------
Title:
Category 1 0.455
Category 2 0.425
Category 3 0.102
Category 4 0.010
Category 5 0.008

Body:
Category 1 0.462
Category 2 0.449
Category 3 0.081
Category 4 0.006
Category 5 0.002


As a human, I can fairly easily analyze the results and see which category(s) should be assigned, whether no category is a strong match, etc.

The question is, how to do this in an automated way?

I almost feel like I should pass the results back through another prediction engine to see whether a particular category should be assigned or not. Or at run a large number of tests with different parameters to see which produces the most accurate results. Parameters for the weight of the title and description probabilities seem like obvious candidates. I have a very large training dataset and also a large testing dataset that was not used in the training.

Advice on the best way to handle this, and/or what is this particular problem called if it has a name?

Edit:

This problem seems to be called multi-label classification. There is a very comprehensive PDF on the topic. It contains a list of "thresholding strategies" that can be used. The authors have created an addon for Weka called Mulan that implements a number of multi-label learning and thresholding strategies.

At this point I need to either try Weka and Mulan, or implement one of the thresholding strategies using the results from my current NLP tool.

• I don't get -- if you have the probability, you simply use the top one, and then you check if the significance is high enough for your needs (i.e. how sure you would like to be the classification is OK). – greenoldman Sep 21 '11 at 10:44
• You posted some probabilities associated with 'Title' and 'Body', each set of 10 probabilities is making reference to a document? – deps_stats Oct 21 '11 at 19:32
• Yes, each result set is for one document. I'll edit the description to make that clearer. – Animism Nov 24 '11 at 21:04