TLDR: What is the best practice in multilabel classification to address the fact that the training and test sets probably won't have completely overlapping sets of labels, and how should I program this in scikit-learn?

This question comes out of my previous question, about whether preprocessing should be applied separately to the training/test sets, or whether it should be applied all at once before splitting into training/test. The consensus in that question was that it should be applied separately.

I am performing multilabel classification (with scikit-learn), using OneVsRestClassifier as a wrapper over LogisticRegression. By using the OneVsRestClassifier I can get multilabel classification.

Before I train OneVsRestClassifier on my data, I need to convert my y_train values from their natural label form [[a, b], [c, a], [a, b, c], ...] to a binary matrix, one row per sample, with each column equaling 1 if the sample has that label, and 0 otherwise. I do this using MultiLabelBinarizer

However, there is a problem. It seems that when I call score(X_test, Y_test), the classifier wants Y_test to have the same number of columns as the Y_train that I previously used in training. This will not happen naturally if I train two copies of MultiLabelBinarizer (one for train, the other for test) because the train and test sets do not have completely overlapping labels.

I have found two solutions but neither is great:

1. Only train one copy of MultiLabelBinarizer, so that all of my target values (for train and test) end up in one big matrix. Then split the matrix when I split between train and test sets. If I do this, I get lots of complaints during the actual model training, because the model notices that certain labels are always 0 in the training examples. These are the labels that are only in the test samples. Other than the warnings, things do end up working.

2. Train MultiLabelBinarizer once, using fit_transform(), on only the training targets. Then just do a simple transform() on the test targets. Since I am not "fitting" again, MultiLabelBinarizer should stick to the original columns that it learned. I believe it is doing this, but unfortunately it won't simply ignore any labels that it hasn't seen before. It just throws a key error, and there don't seem to be any parameters to force it to silently ignore such labels instead.

Of course, with the second solution I am assuming that my test would still be valid even if I am throwing away labels just because they weren't seen in training. My guess is that we would say "a few such cases are OK, but if there are many such cases you may want to reduce them by expanding your training set."

TLDR: What is the best practice in multilabel classification to address the fact that the training and test sets probably won't have completely overlapping sets of labels, and how should I program this in scikit-learn?

This question comes out of my previous question, about whether preprocessing should be applied separately to the training/test sets, or whether it should be applied all at once before splitting into training/test. The consensus in that question was that it should be applied separately.

I am performing multilabel classification (with scikit-learn), using OneVsRestClassifier as a wrapper over LogisticRegression. By using the OneVsRestClassifier I can get multilabel classification.

Before I train OneVsRestClassifier on my data, I need to convert my y_train values from their natural label form [[a, b], [c, a], [a, b, c], ...] to a binary matrix, one row per sample, with each column equaling 1 if the sample has that label, and 0 otherwise. I do this using MultiLabelBinarizer

However, there is a problem. It seems that when I call score(X_test, Y_test), the classifier wants Y_test to have the same number of columns as the Y_train that I previously used in training. This will not happen naturally if I train two copies of MultiLabelBinarizer (one for train, the other for test) because the train and test sets do not have completely overlapping labels.

I have found two solutions but neither is great:

1. Only train one copy of MultiLabelBinarizer, so that all of my target values (for train and test) end up in one big matrix. Then split the matrix when I split between train and test sets. If I do this, I get lots of complaints during the actual model training, because the model notices that certain labels are always 0 in the training examples. These are the labels that are only in the test samples. Other than the warnings, things do end up working.

2. Train MultiLabelBinarizer once, using fit_transform(), on only the training targets. Then just do a simple transform() on the test targets, after manually removing any labels that weren't in the training set.

The second solution has a couple of problems:

A. This could cause some test samples not to have any labels at all. So then those samples also have to be excluded.

B. I have to assume that my test would still be valid even if I am throwing away labels just because they weren't seen in training. My guess is that we would say "a few such cases are OK, but if there are many such cases you may want to reduce them by expanding your training set."

TLDR: What is the best practice in multilabel classification to address the fact that the training and test sets probably won't have completely overlapping sets of labels, and how should I program this in scikit-learn?

This question comes out of my previous question, about whether preprocessing should be applied separately to the training/test sets, or whether it should be applied all at once before splitting into training/test. The consensus in that question was that it should be applied separately.

I am performing multilabel classification (with scikit-learn), using OneVsRestClassifier as a wrapper over LogisticRegression. By using the OneVsRestClassifier I can get multilabel classification.

Before I train OneVsRestClassifier on my data, I need to convert my y_train values from their natural label form ((a, b), (c, a), (a, b, c), ...) to a binary matrix, one row per sample, with each column equaling 1 if the sample has that label, and 0 otherwise. I do this using MultiLabelBinarizer

However, there is a problem. It seems that when I call score(X_test, Y_test), the classifier wants Y_test to have the same number of columns as the Y_train that I previously used in training. This will not happen naturally if I train two copies of MultiLabelBinarizer (one for train, the other for test) because the train and test sets do not have completely overlapping labels.

I have found two solutions but neither is great:

1. Only train one copy of MultiLabelBinarizer, so that all of my target values (for train and test) end up in one big matrix. Then split the matrix when I split between train and test sets. If I do this, I get lots of complaints during the actual model training, because the model notices that certain labels are always 0 in the training examples. These are the labels that are only in the test samples. Other than the warnings, things do end up working.

2. Train MultiLabelBinarizer once, using fit_transform(), on only the training targets. Then just do a simple transform() on the test targets. Since I am not "fitting" again, MultiLabelBinarizer should stick to the original columns that it learned. I believe it is doing this, but unfortunately it won't simply ignore any labels that it hasn't seen before. It just throws a key error, and there don't seem to be any parameters to force it to silently ignore such labels instead.

Of course, with the second solution I am assuming that my test would still be valid even if I am throwing away labels just because they weren't seen in training. My guess is that we would say "a few such cases are OK, but if there are many such cases you may want to reduce them by expanding your training set."

TLDR: What is the best practice in multilabel classification to address the fact that the training and test sets probably won't have completely overlapping sets of labels, and how should I program this in scikit-learn?

This question comes out of my previous question, about whether preprocessing should be applied separately to the training/test sets, or whether it should be applied all at once before splitting into training/test. The consensus in that question was that it should be applied separately.

I am performing multilabel classification (with scikit-learn), using OneVsRestClassifier as a wrapper over LogisticRegression. By using the OneVsRestClassifier I can get multilabel classification.

Before I train OneVsRestClassifier on my data, I need to convert my y_train values from their natural label form [[a, b], [c, a], [a, b, c], ...] to a binary matrix, one row per sample, with each column equaling 1 if the sample has that label, and 0 otherwise. I do this using MultiLabelBinarizer

However, there is a problem. It seems that when I call score(X_test, Y_test), the classifier wants Y_test to have the same number of columns as the Y_train that I previously used in training. This will not happen naturally if I train two copies of MultiLabelBinarizer (one for train, the other for test) because the train and test sets do not have completely overlapping labels.

I have found two solutions but neither is great:

1. Only train one copy of MultiLabelBinarizer, so that all of my target values (for train and test) end up in one big matrix. Then split the matrix when I split between train and test sets. If I do this, I get lots of complaints during the actual model training, because the model notices that certain labels are always 0 in the training examples. These are the labels that are only in the test samples. Other than the warnings, things do end up working.

2. Train MultiLabelBinarizer once, using fit_transform(), on only the training targets. Then just do a simple transform() on the test targets. Since I am not "fitting" again, MultiLabelBinarizer should stick to the original columns that it learned. I believe it is doing this, but unfortunately it won't simply ignore any labels that it hasn't seen before. It just throws a key error, and there don't seem to be any parameters to force it to silently ignore such labels instead.

Of course, with the second solution I am assuming that my test would still be valid even if I am throwing away labels just because they weren't seen in training. My guess is that we would say "a few such cases are OK, but if there are many such cases you may want to reduce them by expanding your training set."