I'm a programmer without a statistics background. I've been working with NLP lately to classify documents, and I'm pretty up to speed with NLP concepts. I've gotten to the point where the NLP software is returning classification probabilities for each category, and the probabilities are quite accurate.
The next task is how to use the result probabilities to assign category(s) to each unlabeled document. Specifically, my dataset has:
The categorization done for each is:
- 30 categories, where each document must belong to one category, and at most two categories. If no category is a strong match, the document is assigned to an "Unknown" category.
- 10 other categories, where each document is only associated with a category if there is a strong match, and each document can belong to as many categories as match.
- 4 other categories, where each document must belong to only one category, and if there isn't a strong match the document is assigned to a default category.
I'm classifying the Title and Body separately. The Body produces the most accurate classifications, but the Title can help improve accuracy in some cases. Examples of result probabilities might be:
Document 1 ---------- Title: Category 1 0.950 Category 2 0.030 Category 3 0.020 Category 4 0.005 Category 5 0.005 Body: Category 1 0.920 Category 2 0.050 Category 3 0.020 Category 4 0.020 Category 5 0.010
Document 2 ---------- Title: Category 1 0.572 Category 2 0.185 Category 3 0.092 Category 4 0.077 Category 5 0.074 Body: Category 1 0.129 Category 2 0.785 Category 3 0.052 Category 4 0.032 Category 5 0.002
Document 3 ---------- Title: Category 1 0.455 Category 2 0.425 Category 3 0.102 Category 4 0.010 Category 5 0.008 Body: Category 1 0.462 Category 2 0.449 Category 3 0.081 Category 4 0.006 Category 5 0.002
As a human, I can fairly easily analyze the results and see which category(s) should be assigned, whether no category is a strong match, etc.
The question is, how to do this in an automated way?
I almost feel like I should pass the results back through another prediction engine to see whether a particular category should be assigned or not. Or at run a large number of tests with different parameters to see which produces the most accurate results. Parameters for the weight of the title and description probabilities seem like obvious candidates. I have a very large training dataset and also a large testing dataset that was not used in the training.
Advice on the best way to handle this, and/or what is this particular problem called if it has a name?
This problem seems to be called multi-label classification. There is a very comprehensive PDF on the topic. It contains a list of "thresholding strategies" that can be used. The authors have created an addon for Weka called Mulan that implements a number of multi-label learning and thresholding strategies.
At this point I need to either try Weka and Mulan, or implement one of the thresholding strategies using the results from my current NLP tool.