achieving consistency between training/test target representations, in multilabel classification

Question

TLDR: What is the best practice in multilabel classification to address the fact that the training and test sets probably won't have completely overlapping sets of labels, and how should I program this in scikit-learn?

This question comes out of my previous question, about whether preprocessing should be applied separately to the training/test sets, or whether it should be applied all at once before splitting into training/test. The consensus in that question was that it should be applied separately.

I am performing multilabel classification (with scikit-learn), using OneVsRestClassifier as a wrapper over LogisticRegression. By using the OneVsRestClassifier I can get multilabel classification.

Before I train OneVsRestClassifier on my data, I need to convert my y_train values from their natural label form [[a, b], [c, a], [a, b, c], ...] to a binary matrix, one row per sample, with each column equaling 1 if the sample has that label, and 0 otherwise. I do this using MultiLabelBinarizer

However, there is a problem. It seems that when I call score(X_test, Y_test), the classifier wants Y_test to have the same number of columns as the Y_train that I previously used in training. This will not happen naturally if I train two copies of MultiLabelBinarizer (one for train, the other for test) because the train and test sets do not have completely overlapping labels.

I have found two solutions but neither is great:

1. Only train one copy of MultiLabelBinarizer, so that all of my target values (for train and test) end up in one big matrix. Then split the matrix when I split between train and test sets. If I do this, I get lots of complaints during the actual model training, because the model notices that certain labels are always 0 in the training examples. These are the labels that are only in the test samples. Other than the warnings, things do end up working.

2. Train MultiLabelBinarizer once, using fit_transform(), on only the training targets. Then just do a simple transform() on the test targets, after manually removing any labels that weren't in the training set.

The second solution has a couple of problems:

A. This could cause some test samples not to have any labels at all. So then those samples also have to be excluded.

B. I have to assume that my test would still be valid even if I am throwing away labels just because they weren't seen in training. My guess is that we would say "a few such cases are OK, but if there are many such cases you may want to reduce them by expanding your training set."

Answer 1

Sorry for the late response.

There are two places where this issue has an effect: training-time and evaluation time. I will try to cover both, below.

I have a similar multi-label classification problem I am working on, also using MultiLabelBinarizer. I am currently using approach number 2 from your list where I fit_transform() once on the training set and then just fit() on the test set.

I am training the classifier to output a catch-all class which should lesson the situation in your "problem A" of having some test samples with no class label identified by the classifier. But your case, I'm not sure those samples need to be excluded from your test. In fact, excluding them will make your metrics less accurate. What I do is compute an accuracy metric per class label such as precision, recall, and F-measure. My final, high-level evaluation metric is a macro-averaged F-measure, averaged over each class's F-measure. Those samples with missing labels would simply contribute false negatives to the accuracy metric -- not ideal, but still reasonably representative of the classifier's true abilities, which is the purpose of the evaluation. So, regarding the evaluation-time issue, I don't think there is a problem as long as you do not throw away samples.

Regarding the training-time issue, this is probably unavoidable. The challenge of creating a training set is always in making it representative of future, unseen samples. The test set is designed to be different from the training set exactly for the purpose of uncovering problems like this before the model is put into production. I see two solutions: (1) increase your training set size so it is more likely to include all classes or (2) replace small classes with a catch-all class. You would use a count or frequency threshold that you adjust up or down in order to balance the trade off between removing too few and too many classes.