Would a sorted class harm a 10-split cross validation?

I am using the Java API of Weka to apply a Naive Bayes classification to an .arff file I've created. The (@data part) file has the following format:

0 0 0 0 1 0 ... 0
1 0 0 0 0 1 ... 0
.
.
.
0 0 1 0 0 1 ... 3
.
.
.
0 0 0 0 1 0 ... 5


Where each number belongs to [0,1], except the last one which is the class [0,5].

Considering that I am using a 10-split cross validation, would it be a mistake to have my data in a sorted by class form? Would that lead in taking the test data from the last class only for example?

I am getting the following confusion matrix, which is obviously wrong:

0.0 |0.0 |0.0 |0.0  |0.0  |0.0|
0.0 |0.0 |0.0 |0.0  |0.0  |0.0|
0.0 |0.0 |0.0 |0.0  |0.0  |0.0|
0.0 |0.0 |0.0 |0.0  |0.0  |0.0|
42.0|14.0|15.0|114.0|233.0|7.0|
71.0|16.0|30.0|241.0|86.0 |7.0|


Any ideas why the first 4 classes are only zeroes? My .arff file has examples (not so evenly distributed) from all 6 classes.

EDIT: I shuffled my data and now I am getting a much more rational result.

27.0|2.0|8.0 |24.0 |27.0 |2.0|
4.0 |2.0|0.0 |2.0  |1.0  |0.0|
6.0 |3.0|15.0|16.0 |19.0 |3.0|
29.0|4.0|13.0|326.0|87.0 |33.0|
20.0|5.0|7.0 |37.0 |110.0|6.0|
5.0 |0.0|1.0 |17.0 |8.0  |7.0|


I am using the code I found here. Is there anything wrong with it?

• Any reasonably (sensible) CV code would randomly assign samples to the 10 groups so ordering in the data would be irrelevant. – Gavin Simpson May 30 '14 at 17:18
• Hi Gavin, I edited my question. Could you please take a look? – Michael May 30 '14 at 18:01

tl;dr: You're calling the wrong function! trainCV doesn't randomly partition the data.

Background

Weka has a few different ways to set up cross validation.

The "top-level" function is weka.classifiers.evaluation.Evalution. This class operates like (and presumably implements) the Experimenter GUI. If you do NOT provide a test set (and don't set the "no-cv" option), it will perform a stratified cross validation. The instance order is shuffled (and you can provide a seed for it). This class will take you a set of instances right through to a performance measurement (e.g., accuracy), which would obviate most of the code in your link. Use this class if you can.

Weka also provides "filters" that process sets of instances. There are two cross-fold validation-related filters: the supervised weka.filters.supervised.instance.StratifiedRemoveFolds and the unsupervised version weka.filters.unsupervised.instance.RemovedFolds. These shuffle your data and then create/remove the specified folds. You can provide a seed so that the fold assignments are reproducible across runs, if you like.

Finally, Instance also includes the weka.instance.trainCV/testCV pair of functions. These both come in two flavours. There is a three argument version, which takes a Java Random object and uses it to shuffle the data, and a two argument version which just blindly assigns the first $k/n$ points to the first fold (etc), without shuffling. This is potentially bad, as you've just discovered.

Your code uses the non-shuffling version of weka.instance.trainCV (see the code for CrossValidationSplit in "Step 3" of your code). Since the data are sorted by class, each fold is very unbalanced (even more than your whole data set), which is why your performance was initially terrible and it improves when you shuffle the data.

For best performance, I'd recommend trying a stratified cross validation with shuffling. If you can use the Evalution class, you can get the whole thing done in a handful of lines. Otherwise, check out the StratifiedRemoveFolds function linked above.

• Hi Matt, I updated my question. Take a look if you like. Thanks. – Michael May 30 '14 at 21:49
• You're using the non-shuffling version of trainCV in the middle of Step 3 (in WekaTest.CrossValidationSplit). You should be calling Instance.randomize() beforehand or otherwise shuffling your data first (which is why the preshuffling helps) – Matt Krause May 30 '14 at 23:01

You already have a good explanation that the usual setup for cross validation is random resampling. Let me add some more considerations:

• One further advantage of random resampling for cross validation is that you can draw more than one random split. That is, you can do iterated or repeated cross validation.

• Sorting the dependent variable is a sensible term only for regression (quantitative/metric dependent) or ordered classes. For the usual setup with independent classes, no order is defined.

However, I'd like to point out that are some (niche) applications where data sorted according to the dependent variable is used for cross validation. I know of two such methods, which are called contiguous block and venetian blinds method by Eigenvector. Other names are interleaved or striped splitting.
Note that all this is about regression.

• If you are concerned that for each fold both surrogate model and test data really span the whole range of the dependent variable, then you could go for the venetian blind splitting.
Personally, I usually go for iterated random resampling and rather monitor the distribution of the dependent. With classification, if I'm concerned that the relative frequencies of the classes may deviate too far, I use stratified ra ndom resampling. That is, I randomly draw from each class the pre-set number of cases. I mainly use this with very small classes, where I have only few cases per cross validation fold.

• The contiguous block splitting with sorted dependent variable for regression has IMHO two particularly interesting surrogate models (folds): the first and the last one. With these you can have a look how your model does at extrapolating just outside the training range (aka calibration range in chemometrics). This can be interesting and important information for regression/calibration.

• The website I linked above points out that this type of resampling may be interesting for time-series as well, but there I'd rather go for a moving-window type of validation.
• Also for classification, it may be interesting to see for each class in turn where it ends up if the whole class was unknown during training. But this again answers a completely different question about model performance which is not the usual goal of cross validation.
• That's a neat idea about using the folds for assessing extrapolated vs. interpolated performance! I'm a little confused about the venetian blind sampling--is it just a poor man's version of a stratified cross validation? – Matt Krause Jun 2 '14 at 12:43
• @MattKrause: Thanks for the flowers :-). Poor man's venetian blinds: I think so. But note it does come from traditional calibration/regression settings, where also the number of affordable calibration samples (physical meaning) is low. I'd say it is a very straightforward way of stratifying according to a metric variable, although more sophisticated methods may exist (I don't know any offhand - I'd argue that if this distinction becomes crucial, it is high time to get back to the lab and get more real samples...). – cbeleites unhappy with SX Jun 2 '14 at 18:53

Directly from wikipedia:

In k-fold cross-validation, the original sample is randomly partitioned into k equal size subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data.

What I mean is: your data is getting scrambled anyway, there's no way that the performance of this metric will be affected by your sorting.

I don't know wecka, but you most likely you have a bug on your implementation. That matrix is telling us that it was only tested with either '4' or '5's.

• You're right that cross validation should involve randomly partitioning the data, but that's not what Weka's trainCV function actually does--it expects you to shuffle the data yourself first. – Matt Krause May 30 '14 at 23:39