I work with R and let's say that I have a train set and a test set. I want to test different algorithms (for example neural networks and svm).
I will perform a first 10-folds cross validation on my train set for tuning the neural network.
Then I will perform a 10-folds cross validation on my train set for tuning the svm.
And I will compare the performances of each best model on my test set.
I was wondering if it was theoretically an issue that the 10-folds (randomly built) were not the same in the tuning of both algorithms.
I think that this should not be a problem because the result of the tuning should be robust to the choice of the folds. But it is apparantly not the case (I've read that about knn, with tune.knn of e1071 package in R).
If we have to fix the splits before tuning, do you know how to do so in R ? I haven't found the right option for the tune function of e1071 package.
Is caret a better package relatively to this point ? Since it seems possible to repeat 10-folds cross validation when tuning, I think that might make the results of the tuning more robust and the comparison of different models more legitimate.
Thanks for your insight