Setting up Multi-label Training Set I have several thousand documents that I want to put in to one or many categories (12 categories total) My current plan is to use Python and SciKit-learn to test multiple machine learning algorithms for which is most predictive. 
I am to the point of creating a labeled training set by taking about a thousand of these documents and assigning labels to each. I have been unable to find a good explanation of how to best label these documents for multilabel classification. I am thinking of creating a csv with a column for the document's id number, and a column for each possible category. Then entering a 1 or 0 if it belongs in that category. Or I was thinking of coding each category as 1-12 and then just making one column where each category is listed but comma separated (1,3,6,12). Before I go through with this lengthy process I was looking for some feedback on how to best label text for multilabel classification.
 A: Some machine learning algortihms (kNN, tree based algorithms including random forests, naive Bayes, graph based algorithms with label propagation etc.) can handle multi class classification inherently. They also output probability estimates for each record to belong to each of the classes.
Usually, you would assign only the most probable class per record. In your case, you can assign the k most probable classes. Or you could assign as many classes as there are above a certain probability threshold.
This allows you to do your multi-class, multi-label classification while running the algorithm only once and not 12 times. It is better o have one binary column per class (and maybe also keep the columns per class with the classification probabilities). Having one column with a varying number of data concatenated is a mess in programming, avoid it.
It is not very clear from your post if this is supervised learning. Do you have the correct label(s) per record? The imagenet data-set is for example tested with multiple labels per record even though there is only one correct label. If the correct label is among the top 5 predicted, this counts as a correct classification. If you have multiple corretc labels per record, it will be more complicated to measure the classification performance comparing them to multiple estimated labels.
A: If I understand correctly, one document can have multiple labels. In this case it would be best to represent labels as an $n \times 12$ matrix. Because then you can easily train 12 binary classifiers via scikit-learn. 
You can only use classification with multiple classes in scikit-learn if each document belongs to one class only. In that case, it would be better to have just one column.
