Would a sorted class harm a 10-split cross validation?

Question

I am using the Java API of Weka to apply a Naive Bayes classification to an .arff file I've created. The (@data part) file has the following format:

0 0 0 0 1 0 ... 0
1 0 0 0 0 1 ... 0
.
.
.
0 0 1 0 0 1 ... 3 
.
.
.
0 0 0 0 1 0 ... 5

Where each number belongs to [0,1], except the last one which is the class [0,5].

Considering that I am using a 10-split cross validation, would it be a mistake to have my data in a sorted by class form? Would that lead in taking the test data from the last class only for example?

I am getting the following confusion matrix, which is obviously wrong:

0.0 |0.0 |0.0 |0.0  |0.0  |0.0|  
0.0 |0.0 |0.0 |0.0  |0.0  |0.0|     
0.0 |0.0 |0.0 |0.0  |0.0  |0.0|   
0.0 |0.0 |0.0 |0.0  |0.0  |0.0|  
42.0|14.0|15.0|114.0|233.0|7.0|  
71.0|16.0|30.0|241.0|86.0 |7.0|

Any ideas why the first 4 classes are only zeroes? My .arff file has examples (not so evenly distributed) from all 6 classes.

EDIT: I shuffled my data and now I am getting a much more rational result.

27.0|2.0|8.0 |24.0 |27.0 |2.0|
4.0 |2.0|0.0 |2.0  |1.0  |0.0|
6.0 |3.0|15.0|16.0 |19.0 |3.0|
29.0|4.0|13.0|326.0|87.0 |33.0|
20.0|5.0|7.0 |37.0 |110.0|6.0|
5.0 |0.0|1.0 |17.0 |8.0  |7.0|

I am using the code I found here. Is there anything wrong with it?

Answer 1

tl;dr: You're calling the wrong function! trainCV doesn't randomly partition the data.

Background

Weka has a few different ways to set up cross validation.

The "top-level" function is weka.classifiers.evaluation.Evalution. This class operates like (and presumably implements) the Experimenter GUI. If you do NOT provide a test set (and don't set the "no-cv" option), it will perform a stratified cross validation. The instance order is shuffled (and you can provide a seed for it). This class will take you a set of instances right through to a performance measurement (e.g., accuracy), which would obviate most of the code in your link. Use this class if you can.

Weka also provides "filters" that process sets of instances. There are two cross-fold validation-related filters: the supervised weka.filters.supervised.instance.StratifiedRemoveFolds and the unsupervised version weka.filters.unsupervised.instance.RemovedFolds. These shuffle your data and then create/remove the specified folds. You can provide a seed so that the fold assignments are reproducible across runs, if you like.

Finally, Instance also includes the weka.instance.trainCV/testCV pair of functions. These both come in two flavours. There is a three argument version, which takes a Java Random object and uses it to shuffle the data, and a two argument version which just blindly assigns the first $k/n$ points to the first fold (etc), without shuffling. This is potentially bad, as you've just discovered.

Your Problem

Your code uses the non-shuffling version of weka.instance.trainCV (see the code for CrossValidationSplit in "Step 3" of your code). Since the data are sorted by class, each fold is very unbalanced (even more than your whole data set), which is why your performance was initially terrible and it improves when you shuffle the data.

For best performance, I'd recommend trying a stratified cross validation with shuffling. If you can use the Evalution class, you can get the whole thing done in a handful of lines. Otherwise, check out the StratifiedRemoveFolds function linked above.

Source: the source (linked above)

Answer 2

You already have a good explanation that the usual setup for cross validation is random resampling. Let me add some more considerations:

• One further advantage of random resampling for cross validation is that you can draw more than one random split. That is, you can do iterated or repeated cross validation.

• Sorting the dependent variable is a sensible term only for regression (quantitative/metric dependent) or ordered classes. For the usual setup with independent classes, no order is defined.

However, I'd like to point out that are some (niche) applications where data sorted according to the dependent variable is used for cross validation. I know of two such methods, which are called contiguous block and venetian blinds method by Eigenvector. Other names are interleaved or striped splitting.
Note that all this is about regression.

• If you are concerned that for each fold both surrogate model and test data really span the whole range of the dependent variable, then you could go for the venetian blind splitting.
  Personally, I usually go for iterated random resampling and rather monitor the distribution of the dependent. With classification, if I'm concerned that the relative frequencies of the classes may deviate too far, I use stratified ra ndom resampling. That is, I randomly draw from each class the pre-set number of cases. I mainly use this with very small classes, where I have only few cases per cross validation fold.

• The contiguous block splitting with sorted dependent variable for regression has IMHO two particularly interesting surrogate models (folds): the first and the last one. With these you can have a look how your model does at extrapolating just outside the training range (aka calibration range in chemometrics). This can be interesting and important information for regression/calibration.

  • The website I linked above points out that this type of resampling may be interesting for time-series as well, but there I'd rather go for a moving-window type of validation.
  • Also for classification, it may be interesting to see for each class in turn where it ends up if the whole class was unknown during training. But this again answers a completely different question about model performance which is not the usual goal of cross validation.

Answer 3

Directly from wikipedia:

In k-fold cross-validation, the original sample is randomly partitioned into k equal size subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data.

What I mean is: your data is getting scrambled anyway, there's no way that the performance of this metric will be affected by your sorting.

I don't know wecka, but you most likely you have a bug on your implementation. That matrix is telling us that it was only tested with either '4' or '5's.