Do we have to fix splits before 10-folds cross validation if we want to compare different algorithms?

Question

I work with R and let's say that I have a train set and a test set. I want to test different algorithms (for example neural networks and svm).

I will perform a first 10-folds cross validation on my train set for tuning the neural network.

Then I will perform a 10-folds cross validation on my train set for tuning the svm.

And I will compare the performances of each best model on my test set.

I was wondering if it was theoretically an issue that the 10-folds (randomly built) were not the same in the tuning of both algorithms.

I think that this should not be a problem because the result of the tuning should be robust to the choice of the folds. But it is apparantly not the case (I've read that about knn, with tune.knn of e1071 package in R).

If we have to fix the splits before tuning, do you know how to do so in R ? I haven't found the right option for the tune function of e1071 package.

Is caret a better package relatively to this point ? Since it seems possible to repeat 10-folds cross validation when tuning, I think that might make the results of the tuning more robust and the comparison of different models more legitimate.

Thanks for your insight

Answer 1

The results will be sensitive to the splits, so you should compare models on the same partitioning of the data. Compare these two approaches:

1. Approach 1 will compare two models, but use the same CV partitioning.
2. Approach 2 will compare two models, but the first model will have a different CV partitioning than second.

We'd like to select the best model. The problem with approach 2 is that the difference in performance between the two models will come from two different sources: (a) the differences between the two folds and (b) the differences between the algorithms themselves (say, random forest and logistic regression). If one model out-performs the other, we won't know if that difference in performance is entirely, partially, or not at all due to the differences in the two CV partitions. On the other hand, any difference in performance using approach 1 cannot be due to differences in how the data were partitioned, because the partitions are identical.

To fix the partioning, use cvTools to create your (repeated) CV partitions and store the results.

Answer 2

It certainly helps, but isn't absolutely essential.

The choice of cross-validation splits introduces a source of (uninteresting) variability. Using the same set of splits removes this source of variance, which might increase your ability to detect variability in the performance of different classifiers (if it exists), which is typically much more interesting.

If you can control the splits, then you really ought to--it's an easy way to increase the power of your experiment without doing much additional work. On the other hand, if you already have some difficult-to-replicate results where you forgot to store the splits, you can certainly use that data; just be aware that comparisons based on those results will not be as powerful as they could be. People often assume that the partition-related variance is a lot smaller than the across-classifier variance, though that may not be true especially if your classes are very unbalanced.

As for e1071 specifically, the docs for tune say

Cross-validation randomizes the data set before building the splits which—once created—remain constant during the training process. The splits can be recovered through the train.ind component of the returned object.

You could let tune generate the folds itself for the first algorithm, then use them, via tune.control (with sampling=fixed), to evaluate subsequent ones. Alternately, it looks like tune generates its partition using global prng (via a call to sample), at the very beginning of the function (Line 71 of Tune.R in the source), so you may be able to generate the same folds by resetting the random number generator's seed before each call to tune. This seems a little brittle though. Finally, this is fairly easy to program yourself.

Answer 3

In addition to @Matt Krause's answer:

I'd approach the question from two different sides:

• One of the basic assumptions underlying cross validation is that the models built on the different splits are equal (or at least equivalent). This allows pooling the results from all those splits. If that assumption is met, then the splitting doesn't matter.
  However, in practice it does happen that the splitting introduces non-negligible variance, i.e. models are unstable wrt. slight changes in the training data, so this variation should anyways be checked (even if not optimizing anything).

• Evaluating different classifiers on the same splits means that you can evaluate the comparison in a paired test, which is more powerful than the corresponding unpaired tests: this is why you can detect smaller changes in performance by keeping the splits constant.

Answer 4

Though this has already been answered, if you want some code that will allow you to use the same CV splits in caret for multiple model trainings, you can use the following:

tune_control <- trainControl(
  method = "repeatedcv", 
  repeats = 2,
  number = 5,
  index = createMultiFolds(df$y, k=5, times=2) # assuming your object is df and you are modeling y
)

You can manually check this worked by training two models and comparing the output of:

model$control$index # replace model w/ name of your model

which should print out something like:

List of 10
Fold1.Rep1: int [1:2400] 1 2 3 4 5 7 9 10 11 12 ...
Fold2.Rep1: int [1:2400] 1 2 3 4 5 6 7 8 9 10 ...
Fold3.Rep1: int [1:2400] 2 3 4 6 8 9 10 11 13 14 ...
Fold4.Rep1: int [1:2400] 1 2 5 6 7 8 11 12 16 18 ...
Fold5.Rep1: int [1:2400] 1 3 4 5 6 7 8 9 10 11 ...
Fold1.Rep2: int [1:2400] 1 3 4 5 6 8 10 11 12 14 ...
Fold2.Rep2: int [1:2400] 1 2 3 4 5 6 7 8 9 10 ...
Fold3.Rep2: int [1:2400] 2 3 4 5 7 8 9 10 11 12 ...
Fold4.Rep2: int [1:2400] 1 2 3 5 6 7 8 9 11 12 ...
Fold5.Rep2: int [1:2400] 1 2 4 6 7 9 10 13 16 17 ...