7
$\begingroup$

I work with R and let's say that I have a train set and a test set. I want to test different algorithms (for example neural networks and svm).

I will perform a first 10-folds cross validation on my train set for tuning the neural network.

Then I will perform a 10-folds cross validation on my train set for tuning the svm.

And I will compare the performances of each best model on my test set.

I was wondering if it was theoretically an issue that the 10-folds (randomly built) were not the same in the tuning of both algorithms.

I think that this should not be a problem because the result of the tuning should be robust to the choice of the folds. But it is apparantly not the case (I've read that about knn, with tune.knn of e1071 package in R).

If we have to fix the splits before tuning, do you know how to do so in R ? I haven't found the right option for the tune function of e1071 package.

Is caret a better package relatively to this point ? Since it seems possible to repeat 10-folds cross validation when tuning, I think that might make the results of the tuning more robust and the comparison of different models more legitimate.

Thanks for your insight

Improve this question
$\endgroup$
3
  • $\begingroup$ You really don't need a package to do random K-fold cross-validation - it's very simple to code directly, and this gives you much more flexibility than any available package. First, use sample() to create a random number 1 thru 10 for each row of your data. These numbers represent the CV fold partition. Then use a for loop to do model fitting and testing on each of the train/test splits. To force all models to use the same CV partition, set the partition at the beginning of your script and then train&test all models based on that assignment. $\endgroup$ – Paul Aug 5 '15 at 15:19
  • $\begingroup$ Thank you Paul for your answer. I do know how to create my partitions, I juste want to know 1. is it theoretically acceptable to use different partitions when using different algorithms at the cv step (which allow to use very user-friendly functions like tune in e1071) 2. if not, is it possible to use my own partitions in a function like tune in e1071 (I don't find how and I find it silly to do a loop myself whereas somebody has already implemented all the cross-validation in a properly way). But thanks for your reactivity $\endgroup$ – Stéphanie C Aug 5 '15 at 15:32
  • $\begingroup$ Adding that I am speaking of the cross-validation here and not the final comparison on the test datasets (which of course have to be the same) $\endgroup$ – Stéphanie C Aug 5 '15 at 15:33
2
$\begingroup$

The results will be sensitive to the splits, so you should compare models on the same partitioning of the data. Compare these two approaches:

  1. Approach 1 will compare two models, but use the same CV partitioning.
  2. Approach 2 will compare two models, but the first model will have a different CV partitioning than second.

We'd like to select the best model. The problem with approach 2 is that the difference in performance between the two models will come from two different sources: (a) the differences between the two folds and (b) the differences between the algorithms themselves (say, random forest and logistic regression). If one model out-performs the other, we won't know if that difference in performance is entirely, partially, or not at all due to the differences in the two CV partitions. On the other hand, any difference in performance using approach 1 cannot be due to differences in how the data were partitioned, because the partitions are identical.

To fix the partioning, use cvTools to create your (repeated) CV partitions and store the results.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Sorry I don't see your point, I don't see what can be tuned in a logistic regression routine (unless you speak about variables selection). Tuning is done by 10-folds cross-validation (or other types of cross-validation) in order to tune parameters with robustess relatively to the random sampling, so intuitively, the resulting parameters should not depend too much of the ten folds of the beginning. Or I missing something ? $\endgroup$ – Stéphanie C Aug 5 '15 at 15:20
  • $\begingroup$ I think that the idea is that the difference between the measures of performance on different folds due to randomness of the folds should be smaller than the difference due to a different choice of hyperparameters. Unless I don't see the point of cross-validation for tuning at all... $\endgroup$ – Stéphanie C Aug 5 '15 at 15:40
  • $\begingroup$ Ok I totally agree with you now, and this is precisely my question. Theoretically we should perform some repeated cross-validation for them to have a real sense right ? I will have a look at cvTools thanks. But I would be surprised that such an important point is not treated in e1071 which seems a rather complete library $\endgroup$ – Stéphanie C Aug 5 '15 at 15:42
  • $\begingroup$ @StéphanieC CV is repeated to mitigate sensitivity of the results to any particular partitioning of the data. e1071 is not intended to be a complete library -- it's literally described as "Misc Functions of the Department of Statistics, Probability Theory Group." $\endgroup$ – Sycorax♦ Aug 5 '15 at 15:44
  • $\begingroup$ I am not happy with your answer because I was happy with e1071 and that I have to learn a new package but you must have the correct answer anyway (thanks!) :-). Do you have any preference between caret or cvTools ? I saw that caret offers the possibility to fix partitions or repeat cross validation too $\endgroup$ – Stéphanie C Aug 5 '15 at 18:33
4
$\begingroup$

It certainly helps, but isn't absolutely essential.

The choice of cross-validation splits introduces a source of (uninteresting) variability. Using the same set of splits removes this source of variance, which might increase your ability to detect variability in the performance of different classifiers (if it exists), which is typically much more interesting.

If you can control the splits, then you really ought to--it's an easy way to increase the power of your experiment without doing much additional work. On the other hand, if you already have some difficult-to-replicate results where you forgot to store the splits, you can certainly use that data; just be aware that comparisons based on those results will not be as powerful as they could be. People often assume that the partition-related variance is a lot smaller than the across-classifier variance, though that may not be true especially if your classes are very unbalanced.

As for e1071 specifically, the docs for tune say

Cross-validation randomizes the data set before building the splits which—once created—remain constant during the training process. The splits can be recovered through the train.ind component of the returned object.

You could let tune generate the folds itself for the first algorithm, then use them, via tune.control (with sampling=fixed), to evaluate subsequent ones. Alternately, it looks like tune generates its partition using global prng (via a call to sample), at the very beginning of the function (Line 71 of Tune.R in the source), so you may be able to generate the same folds by resetting the random number generator's seed before each call to tune. This seems a little brittle though. Finally, this is fairly easy to program yourself.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you very much for your answer. I had read that in the e1071 documentation, and this is cool that you can store the index of your folds but no that much useful if you can't reinject them in another tune wrapper. Maybe set seed before and then use the tune.control with sampling fixed for all the algorithms will be enough (and simple !). I will try that right away $\endgroup$ – Stéphanie C Aug 5 '15 at 17:46
  • $\begingroup$ I might misunderstand but actually I don't think that tune.control(sampling="fix") allows to make cross-validation on a fixed partition... $\endgroup$ – Stéphanie C Aug 5 '15 at 18:05
  • $\begingroup$ It just gets you one run with a fixed train/test set. You'd have to call it yourself in a loop to do the whole 10-fold CV. $\endgroup$ – Matt Krause Aug 5 '15 at 18:12
  • $\begingroup$ well, too bad then ! but thank you anyway for your answer $\endgroup$ – Stéphanie C Aug 5 '15 at 18:31
2
$\begingroup$

In addition to @Matt Krause's answer:

I'd approach the question from two different sides:

  • One of the basic assumptions underlying cross validation is that the models built on the different splits are equal (or at least equivalent). This allows pooling the results from all those splits. If that assumption is met, then the splitting doesn't matter.
    However, in practice it does happen that the splitting introduces non-negligible variance, i.e. models are unstable wrt. slight changes in the training data, so this variation should anyways be checked (even if not optimizing anything).

  • Evaluating different classifiers on the same splits means that you can evaluate the comparison in a paired test, which is more powerful than the corresponding unpaired tests: this is why you can detect smaller changes in performance by keeping the splits constant.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer. I may add that the way you split may entail more or less variance, I mean if you split in more folds, the intersection between two train sets for cross validation is bigger, so is there a rule of the thumb for choosing between 20-folds, 10-folds or 5-folds, apart from computing considerations and cases where we have multiple unbalanced classes ? $\endgroup$ – Stéphanie C Aug 6 '15 at 13:00
  • $\begingroup$ See e.g. stats.stackexchange.com/questions/27730/…. Usually, this doesn't matter that much (if models built on 90% of the cases are noticeably unstable, typically models built on 95% of the cases are not that much more stable). $\endgroup$ – cbeleites unhappy with SX Aug 6 '15 at 13:07
0
$\begingroup$

Though this has already been answered, if you want some code that will allow you to use the same CV splits in caret for multiple model trainings, you can use the following: 

tune_control <- trainControl(
  method = "repeatedcv", 
  repeats = 2,
  number = 5,
  index = createMultiFolds(df$y, k=5, times=2) # assuming your object is df and you are modeling y
)

You can manually check this worked by training two models and comparing the output of:

model$control$index # replace model w/ name of your model

which should print out something like:

List of 10
Fold1.Rep1: int [1:2400] 1 2 3 4 5 7 9 10 11 12 ...
Fold2.Rep1: int [1:2400] 1 2 3 4 5 6 7 8 9 10 ...
Fold3.Rep1: int [1:2400] 2 3 4 6 8 9 10 11 13 14 ...
Fold4.Rep1: int [1:2400] 1 2 5 6 7 8 11 12 16 18 ...
Fold5.Rep1: int [1:2400] 1 3 4 5 6 7 8 9 10 11 ...
Fold1.Rep2: int [1:2400] 1 3 4 5 6 8 10 11 12 14 ...
Fold2.Rep2: int [1:2400] 1 2 3 4 5 6 7 8 9 10 ...
Fold3.Rep2: int [1:2400] 2 3 4 5 7 8 9 10 11 12 ...
Fold4.Rep2: int [1:2400] 1 2 3 5 6 7 8 9 11 12 ...
Fold5.Rep2: int [1:2400] 1 2 4 6 7 9 10 13 16 17 ...
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.