# How to use scikit-learn's cross validation functions on multi-label classifiers

I'm testing different classifiers on a data set where there are 5 classes and each instance can belong to one or more of these classes, so I'm using scikit-learn's multi-label classifiers, specifically sklearn.multiclass.OneVsRestClassifier. Now I want to perform cross-validation using the sklearn.cross_validation.StratifiedKFold. This produces the following error:

Traceback (most recent call last):
File "mlfromcsv.py", line 93, in <module>
main()
File "mlfromcsv.py", line 77, in main
test_classifier_multilabel(svm.LinearSVC(), X, Y, 'Linear Support Vector Machine')
File "mlfromcsv.py", line 44, in test_classifier_multilabel
scores = cross_validation.cross_val_score(clf_ml, X, Y_list, cv=cv, score_func=metrics.precision_recall_fscore_support, n_jobs=jobs)
File "/usr/lib/pymodules/python2.7/sklearn/cross_validation.py", line 1046, in cross_val_score
X, y = check_arrays(X, y, sparse_format='csr')
File "/usr/lib/pymodules/python2.7/sklearn/utils/validation.py", line 144, in check_arrays
size, n_samples))
ValueError: Found array with dim 5. Expected 98816


Note that training the multi-label classifier does not crash, but the cross-validation does. How must I perform cross-validation for this multi-label classifier?

I have also written a second version that breaks down the problem into training and cross-validating 5 separate classifiers. This works just fine.

Here is my code. The function test_classifier_multilabel is the one giving problems. test_classifier is my other attempt (breaking up the problem into 5 classifiers and 5 cross-validations).

import numpy as np
from sklearn import *
from sklearn.multiclass import OneVsRestClassifier
from sklearn.neighbors import KNeighborsClassifier
import time

def test_classifier(clf, X, Y, description, jobs=1):
print '=== Testing classifier {0} ==='.format(description)
for class_idx in xrange(Y.shape[1]):
print ' > Cross-validating for class {:d}'.format(class_idx)
n_samples = X.shape[0]
cv = cross_validation.StratifiedKFold(Y[:,class_idx], 3)
t_start = time.clock()
scores = cross_validation.cross_val_score(clf, X, Y[:,class_idx], cv=cv, score_func=metrics.precision_recall_fscore_support, n_jobs=jobs)
t_end = time.clock();
print 'Cross validation time: {:0.3f}s.'.format(t_end-t_start)
str_tbl_fmt = '{:>15s}{:>15s}{:>15s}{:>15s}{:>15s}'
str_tbl_entry_fmt = '{:0.2f} +/- {:0.2f}'
print str_tbl_fmt.format('', 'Precision', 'Recall', 'F1 score', 'Support')
for (score_class, lbl) in [(0, 'Negative'), (1, 'Positive')]:
mean_precision = scores[:,0,score_class].mean()
std_precision = scores[:,0,score_class].std()
mean_recall = scores[:,1,score_class].mean()
std_recall = scores[:,1,score_class].std()
mean_f1_score = scores[:,2,score_class].mean()
std_f1_score = scores[:,2,score_class].std()
support = scores[:,3,score_class].mean()
print str_tbl_fmt.format(
lbl,
str_tbl_entry_fmt.format(mean_precision, std_precision),
str_tbl_entry_fmt.format(mean_recall, std_recall),
str_tbl_entry_fmt.format(mean_f1_score, std_f1_score),
'{:0.2f}'.format(support))

def test_classifier_multilabel(clf, X, Y, description, jobs=1):
print '=== Testing multi-label classifier {0} ==='.format(description)
n_samples = X.shape[0]
Y_list = [value for value in Y.T]
print 'Y_list[0].shape:', Y_list[0].shape, 'len(Y_list):', len(Y_list)
cv = cross_validation.StratifiedKFold(Y_list, 3)
clf_ml = OneVsRestClassifier(clf)
accuracy = (clf_ml.fit(X, Y).predict(X) != Y).sum()
print 'Accuracy: {:0.2f}'.format(accuracy)
scores = cross_validation.cross_val_score(clf_ml, X, Y_list, cv=cv, score_func=metrics.precision_recall_fscore_support, n_jobs=jobs)
str_tbl_fmt = '{:>15s}{:>15s}{:>15s}{:>15s}{:>15s}'
str_tbl_entry_fmt = '{:0.2f} +/- {:0.2f}'
print str_tbl_fmt.format('', 'Precision', 'Recall', 'F1 score', 'Support')
for (score_class, lbl) in [(0, 'Negative'), (1, 'Positive')]:
mean_precision = scores[:,0,score_class].mean()
std_precision = scores[:,0,score_class].std()
mean_recall = scores[:,1,score_class].mean()
std_recall = scores[:,1,score_class].std()
mean_f1_score = scores[:,2,score_class].mean()
std_f1_score = scores[:,2,score_class].std()
support = scores[:,3,score_class].mean()
print str_tbl_fmt.format(
lbl,
str_tbl_entry_fmt.format(mean_precision, std_precision),
str_tbl_entry_fmt.format(mean_recall, std_recall),
str_tbl_entry_fmt.format(mean_f1_score, std_f1_score),
'{:0.2f}'.format(support))

def main():
nfeatures = 13
nclasses = 5
ncolumns = nfeatures + nclasses

data = np.loadtxt('./feature_db.csv', delimiter=',', usecols=range(ncolumns))

print data, data.shape
X = np.hstack((data[:,0:3], data[:,(nfeatures-1):nfeatures]))
print 'X.shape:', X.shape
Y = data[:,nfeatures:ncolumns]
print 'Y.shape:', Y.shape

test_classifier(svm.LinearSVC(), X, Y, 'Linear Support Vector Machine', jobs=-1)
test_classifier_multilabel(svm.LinearSVC(), X, Y, 'Linear Support Vector Machine')

if  __name__ =='__main__':
main()


I am using Ubuntu 13.04 and scikit-learn 0.12. My data is in the form of two arrays (X and Y) that have shapes (98816, 4) and (98816, 5), i.e. 4 features per instance and 5 class labels. The labels are either 1 or 0 to indicated membership within that class. Am I using the correct format as I don't see much documentation about that?

Stratified sampling means that the class membership distribution is preserved in your KFold sampling. This doesn't make a lot of sense in the multilabel case where your target vector might have more than one label per observation.

There are two possible interpretations of stratified in this sense.

For $n$ labels where at least one of them is filled that gives you $\sum\limits_{i=1}^n2^n$ unique labels. You could perform stratified sampling on the each of the unique label bins.

The other option is to try and segment the training data s.t. that probability mass of the distribution of the label vectors is approximately the same over the folds. E.g.

import numpy as np

np.random.seed(1)
y = np.random.randint(0, 2, (5000, 5))
y = y[np.where(y.sum(axis=1) != 0)[0]]

def proba_mass_split(y, folds=7):
obs, classes = y.shape
dist = y.sum(axis=0).astype('float')
dist /= dist.sum()
index_list = []
fold_dist = np.zeros((folds, classes), dtype='float')
for _ in xrange(folds):
index_list.append([])
for i in xrange(obs):
if i < folds:
target_fold = i
else:
normed_folds = fold_dist.T / fold_dist.sum(axis=1)
how_off = normed_folds.T - dist
target_fold = np.argmin(np.dot((y[i] - .5).reshape(1, -1), how_off.T))
fold_dist[target_fold] += y[i]
index_list[target_fold].append(i)
print("Fold distributions are")
print(fold_dist)
return index_list

if __name__ == '__main__':
proba_mass_split(y)


To get the normal training, testing indices that KFold produces you want to rewrite that to it returns the np.setdiff1d of each index with np.arange(y.shape[0]), then wrap that in a class with an iter method.

• Thanks for this explanation. I'd just like to check something, does the OneVsRestClassifier accept a 2D array (e.g. y in your example code) or a tuple of lists of class labels? I ask because I looked at the multi-label classification example on scikit-learn just now and saw that the make_multilabel_classification function returns a tuple of lists of class labels, e.g. ([2], [0], [0, 2], [0]...) when using 3 classes? – chippies Jul 28 '13 at 20:40
• It works both ways. When a list of tuples is passed it fits a sklearn.preprocessing.LabelBinarizer to it. You know a few of the algorithms work in the multiclass multilabel case. Notably RandomForest. – Jessica Mick Jul 28 '13 at 21:43
• Thanks very much, this at least got me past the crashes. For the moment I've switched to K-fold cross validation but I think I will use your code soon. Now however, the score returned by cross_val_score only has two columns, i.e. as if there are only two classes. Changing to metrics.confusion_matrix produces 2x2 confusion matrices. Do any of the metrics support multi-label classifiers? – chippies Jul 29 '13 at 10:11
• I've answered my own sub-question. Metrics that support multi-label classifiers only appeared in scikit-learn 0.14-rc, so I'll have to upgrade if I want that ability, or do it myself. Thanks for the help and code. – chippies Jul 29 '13 at 10:47
• I removed arraying on the return statement. There's no reason that you'll always find a perfectly partitioned set of data points. Let me know if this works out. You should also write some tests in your code. I kind of breathed out this algorithm after staring at convex optimization algorithms all day. – Jessica Mick Jul 30 '13 at 0:04

You might want to check: On the stratification of multi-label data .

Here the authors first tell the simple idea of sampling from unique labelsets and then introduce a new approach iterative stratification for multi-label datasets.

The approach of iterative stratification is greedy.

For a quick overview, here is what the iterative stratification does:

First they find out how many examples should go into each of the k-folds.

• Find the desired number of examples per fold $i$ per label $j$, $c_i^j$ .

• From the dataset which are yet to be distributed into k-folds, the label $l$ is identified for which the number of examples are the minimum, $D^l$ .

• Then for each datapoint in $D^l$ find the fold $k$ for which $c_k^j$ is maximized (break ties here). Which is in other words mean: which fold has the maximum demand for label $l$, or is the most imbalanced with respect to label $l$.

• Add the current datapoint to the fold $k$ found from above step, remove the datapoint from the original dataset and adjust the count values of $c$ and continue until all the datapoints are not distributed into the folds.

The main idea is to first focus on the labels which are rare, this idea comes from the hypothesis that

"if rare labels are not examined in priority, then they may be distributed in an undesired way, and this cannot be repaired subsequently"

To understand how ties are broken and other details, I will recommend reading the paper. Also, from the experiments section what I can understand is, depending on the labelset/examples ratio one might want to use the unique labelset based or this proposed iterative stratification method. For lower values of this ratio the distribution of the labels across the folds are close or better in a few cases as iterative stratification. For higher values of this ratio, iterative stratification is shown to have maintained better distributions in the folds.