1
$\begingroup$

I have several thousand documents that I want to put in to one or many categories (12 categories total) My current plan is to use Python and SciKit-learn to test multiple machine learning algorithms for which is most predictive.

I am to the point of creating a labeled training set by taking about a thousand of these documents and assigning labels to each. I have been unable to find a good explanation of how to best label these documents for multilabel classification. I am thinking of creating a csv with a column for the document's id number, and a column for each possible category. Then entering a 1 or 0 if it belongs in that category. Or I was thinking of coding each category as 1-12 and then just making one column where each category is listed but comma separated (1,3,6,12). Before I go through with this lengthy process I was looking for some feedback on how to best label text for multilabel classification.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ this is not really related to stats or ML. Try maybe the data science SE website. But I don't see why a .csv or space separated .txt file with two columns (one for unique doc ID and one for category from 1-12) wouldn't work $\endgroup$
    – Antoine
    Nov 15 '16 at 20:50
1
$\begingroup$

Some machine learning algortihms (kNN, tree based algorithms including random forests, naive Bayes, graph based algorithms with label propagation etc.) can handle multi class classification inherently. They also output probability estimates for each record to belong to each of the classes.

Usually, you would assign only the most probable class per record. In your case, you can assign the k most probable classes. Or you could assign as many classes as there are above a certain probability threshold.

This allows you to do your multi-class, multi-label classification while running the algorithm only once and not 12 times. It is better o have one binary column per class (and maybe also keep the columns per class with the classification probabilities). Having one column with a varying number of data concatenated is a mess in programming, avoid it.

It is not very clear from your post if this is supervised learning. Do you have the correct label(s) per record? The imagenet data-set is for example tested with multiple labels per record even though there is only one correct label. If the correct label is among the top 5 predicted, this counts as a correct classification. If you have multiple corretc labels per record, it will be more complicated to measure the classification performance comparing them to multiple estimated labels.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for the response. I am leaning toward KNN and random forest. Ideally, I'd like to test both to see which is more predictive. I am in the process of creating the training set for supervised learning. The plan is to go through and assign the correct label to about a thousand documents. This will make up the training set. My question is then how best to structure this training set for multilabel classification. $\endgroup$
    – J.Gorman
    Nov 15 '16 at 20:40
  • $\begingroup$ So you will be the human expert that assigns one correct label per document? Don't forget to also do this for the test-set, otherwise you cannot measure performance on new data. And the multi-label part is only to give the classifiers more chances to have the correct document among its top choices? $\endgroup$
    – David Ernst
    Nov 15 '16 at 20:45
  • $\begingroup$ Yes, I and a few others, will be giving the correct labels. Yes, we will do the same for the test set. I am not sure what you mean in your third sentence. The multi label part is to train the model how to identify multiple labels as some documents will fall into multiple categories. $\endgroup$
    – J.Gorman
    Nov 15 '16 at 22:29
  • $\begingroup$ So then there are multiple correct labels (manually assigned) and multiple predicted labels (for which the algorithm needs to run only once though). Will you manually assign the same number of correct labels per record? If not, the algorithm would also need to guess how many correct labels there are (in addition to what those labels are). The algorithm has more chances of making mistakes in this scenario and you need to search of an adequate way to measure performance. $\endgroup$
    – David Ernst
    Nov 16 '16 at 0:19
  • $\begingroup$ If you count the classification as correct only if the algorithm finds the right number of labels and finds all the correct labels (perhaps even in the right order of likeliness), you will have very low accuracy because this is very hard. $\endgroup$
    – David Ernst
    Nov 16 '16 at 0:21
0
$\begingroup$

If I understand correctly, one document can have multiple labels. In this case it would be best to represent labels as an $n \times 12$ matrix. Because then you can easily train 12 binary classifiers via scikit-learn.

You can only use classification with multiple classes in scikit-learn if each document belongs to one class only. In that case, it would be better to have just one column.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you. Would it be a times 12 matrix or times 12+1? I am planning on including the document id in the matrix as well as the binary classifiers. $\endgroup$
    – J.Gorman
    Nov 15 '16 at 20:37
  • $\begingroup$ Oh yeah it would be 13 columns $\endgroup$
    – erensezener
    Nov 15 '16 at 22:11
  • $\begingroup$ This depends on your programming language. You could either make sure that the line order in the output matrix is the same as in the input data matrix. Or you could add an explicit doc id column in which case the order may differ at some point, since you can sort the lines back to the right state. Or you could keep input and output data in the same matrix (and make sure you only hand the range of the input columns to the algorithm)... $\endgroup$
    – David Ernst
    Nov 15 '16 at 22:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.