How to use scikit-learn's cross validation functions on multi-label classifiers I'm testing different classifiers on a data set where there are 5 classes and each instance can belong to one or more of these classes, so I'm using scikit-learn's multi-label classifiers, specifically sklearn.multiclass.OneVsRestClassifier.  Now I want to perform cross-validation using the sklearn.cross_validation.StratifiedKFold.  This produces the following error:
Traceback (most recent call last):
  File "mlfromcsv.py", line 93, in <module>
    main()
  File "mlfromcsv.py", line 77, in main
    test_classifier_multilabel(svm.LinearSVC(), X, Y, 'Linear Support Vector Machine')
  File "mlfromcsv.py", line 44, in test_classifier_multilabel
    scores = cross_validation.cross_val_score(clf_ml, X, Y_list, cv=cv, score_func=metrics.precision_recall_fscore_support, n_jobs=jobs)
  File "/usr/lib/pymodules/python2.7/sklearn/cross_validation.py", line 1046, in cross_val_score
    X, y = check_arrays(X, y, sparse_format='csr')
  File "/usr/lib/pymodules/python2.7/sklearn/utils/validation.py", line 144, in check_arrays
    size, n_samples))
ValueError: Found array with dim 5. Expected 98816

Note that training the multi-label classifier does not crash, but the cross-validation does.  How must I perform cross-validation for this multi-label classifier?
I have also written a second version that breaks down the problem into training and cross-validating 5 separate classifiers.  This works just fine.
Here is my code.  The function test_classifier_multilabel is the one giving problems.  test_classifier is my other attempt (breaking up the problem into 5 classifiers and 5 cross-validations).
import numpy as np
from sklearn import *
from sklearn.multiclass import OneVsRestClassifier
from sklearn.neighbors import KNeighborsClassifier
import time

def test_classifier(clf, X, Y, description, jobs=1):
    print '=== Testing classifier {0} ==='.format(description)
    for class_idx in xrange(Y.shape[1]):
        print ' > Cross-validating for class {:d}'.format(class_idx)
        n_samples = X.shape[0]
        cv = cross_validation.StratifiedKFold(Y[:,class_idx], 3)
        t_start = time.clock()
        scores = cross_validation.cross_val_score(clf, X, Y[:,class_idx], cv=cv, score_func=metrics.precision_recall_fscore_support, n_jobs=jobs)
        t_end = time.clock();
        print 'Cross validation time: {:0.3f}s.'.format(t_end-t_start)
        str_tbl_fmt = '{:>15s}{:>15s}{:>15s}{:>15s}{:>15s}'
        str_tbl_entry_fmt = '{:0.2f} +/- {:0.2f}'
        print str_tbl_fmt.format('', 'Precision', 'Recall', 'F1 score', 'Support')
        for (score_class, lbl) in [(0, 'Negative'), (1, 'Positive')]:
            mean_precision = scores[:,0,score_class].mean()
            std_precision = scores[:,0,score_class].std()
            mean_recall = scores[:,1,score_class].mean()
            std_recall = scores[:,1,score_class].std()
            mean_f1_score = scores[:,2,score_class].mean()
            std_f1_score = scores[:,2,score_class].std()
            support = scores[:,3,score_class].mean()
            print str_tbl_fmt.format(
                lbl,
                str_tbl_entry_fmt.format(mean_precision, std_precision),
                str_tbl_entry_fmt.format(mean_recall, std_recall),
                str_tbl_entry_fmt.format(mean_f1_score, std_f1_score),
                '{:0.2f}'.format(support))

def test_classifier_multilabel(clf, X, Y, description, jobs=1):
    print '=== Testing multi-label classifier {0} ==='.format(description)
    n_samples = X.shape[0]
    Y_list = [value for value in Y.T]
    print 'Y_list[0].shape:', Y_list[0].shape, 'len(Y_list):', len(Y_list)
    cv = cross_validation.StratifiedKFold(Y_list, 3)
    clf_ml = OneVsRestClassifier(clf)
    accuracy = (clf_ml.fit(X, Y).predict(X) != Y).sum()
    print 'Accuracy: {:0.2f}'.format(accuracy)
    scores = cross_validation.cross_val_score(clf_ml, X, Y_list, cv=cv, score_func=metrics.precision_recall_fscore_support, n_jobs=jobs)
    str_tbl_fmt = '{:>15s}{:>15s}{:>15s}{:>15s}{:>15s}'
    str_tbl_entry_fmt = '{:0.2f} +/- {:0.2f}'
    print str_tbl_fmt.format('', 'Precision', 'Recall', 'F1 score', 'Support')
    for (score_class, lbl) in [(0, 'Negative'), (1, 'Positive')]:
        mean_precision = scores[:,0,score_class].mean()
        std_precision = scores[:,0,score_class].std()
        mean_recall = scores[:,1,score_class].mean()
        std_recall = scores[:,1,score_class].std()
        mean_f1_score = scores[:,2,score_class].mean()
        std_f1_score = scores[:,2,score_class].std()
        support = scores[:,3,score_class].mean()
        print str_tbl_fmt.format(
            lbl,
            str_tbl_entry_fmt.format(mean_precision, std_precision),
            str_tbl_entry_fmt.format(mean_recall, std_recall),
            str_tbl_entry_fmt.format(mean_f1_score, std_f1_score),
            '{:0.2f}'.format(support))

def main():
    nfeatures = 13
    nclasses = 5
    ncolumns = nfeatures + nclasses

    data = np.loadtxt('./feature_db.csv', delimiter=',', usecols=range(ncolumns))

    print data, data.shape
    X = np.hstack((data[:,0:3], data[:,(nfeatures-1):nfeatures]))
    print 'X.shape:', X.shape
    Y = data[:,nfeatures:ncolumns]
    print 'Y.shape:', Y.shape

    test_classifier(svm.LinearSVC(), X, Y, 'Linear Support Vector Machine', jobs=-1)
    test_classifier_multilabel(svm.LinearSVC(), X, Y, 'Linear Support Vector Machine')

if  __name__ =='__main__':
    main()

I am using Ubuntu 13.04 and scikit-learn 0.12.  My data is in the form of two arrays (X and Y) that have shapes (98816, 4) and (98816, 5), i.e. 4 features per instance and 5 class labels.  The labels are either 1 or 0 to indicated membership within that class.  Am I using the correct format as I don't see much documentation about that?
 A: You might want to check: On the stratification of multi-label data .
Here the authors first tell the simple idea of sampling from unique labelsets and then introduce a new approach iterative stratification for multi-label datasets.
The approach of iterative stratification is greedy.
For a quick overview, here is what the iterative stratification does:
First they find out how many examples should go into each of the k-folds. 


*

*Find the desired number of examples per fold $i$ per label $j$, $c_i^j$ .

*From the dataset which are yet to be distributed into k-folds, the label $l$ is identified for which the number of examples are the minimum, $D^l$ .

*Then for each datapoint in $D^l$ find the fold $k$ for which $c_k^j$ is maximized (break ties here). Which is in other words mean: which fold has the maximum demand for label $l$, or is the most imbalanced with respect to label $l$.

*Add the current datapoint to the fold $k$ found from above step, remove the datapoint from the original dataset and adjust the count values of $c$ and continue until all the datapoints are not distributed into the folds.
The main idea is to first focus on the labels which are rare, this idea comes from the hypothesis that 

"if rare labels are not examined in priority, then they may be
  distributed in an undesired way, and this cannot be repaired
  subsequently"

To understand how ties are broken and other details, I will recommend reading the paper. Also, from the experiments section what I can understand is, depending on the labelset/examples ratio one might want to use the unique labelset based or this proposed iterative stratification method. For lower values of this ratio the distribution of the labels across the folds are close or better in a few cases as iterative stratification. For higher values of this ratio, iterative stratification is shown to have maintained better distributions in the folds.
A: Stratified sampling means that the class membership distribution is preserved in your KFold sampling. This doesn't make a lot of sense in the multilabel case where your target vector might have more than one label per observation.
There are two possible interpretations of stratified in this sense.
For $n$ labels where at least one of them is filled that gives you $\sum\limits_{i=1}^n2^n$ unique labels. You could perform stratified sampling on the each of the unique label bins.
The other option is to try and segment the training data s.t. that probability mass of the distribution of the label vectors is approximately the same over the folds. E.g.
import numpy as np

np.random.seed(1)
y = np.random.randint(0, 2, (5000, 5))
y = y[np.where(y.sum(axis=1) != 0)[0]]


def proba_mass_split(y, folds=7):
    obs, classes = y.shape
    dist = y.sum(axis=0).astype('float')
    dist /= dist.sum()
    index_list = []
    fold_dist = np.zeros((folds, classes), dtype='float')
    for _ in xrange(folds):
        index_list.append([])
    for i in xrange(obs):
        if i < folds:
            target_fold = i
        else:
            normed_folds = fold_dist.T / fold_dist.sum(axis=1)
            how_off = normed_folds.T - dist
            target_fold = np.argmin(np.dot((y[i] - .5).reshape(1, -1), how_off.T))
        fold_dist[target_fold] += y[i]
        index_list[target_fold].append(i)
    print("Fold distributions are")
    print(fold_dist)
    return index_list

if __name__ == '__main__':
    proba_mass_split(y)

To get the normal training, testing indices that KFold produces you want to rewrite that to it returns the np.setdiff1d of each index with np.arange(y.shape[0]), then wrap that in a class with an iter method.
