Parameter selection and k-fold cross validation

Question

I have one dataset, and need to do cross-validation, for example, a 10-fold cross-validation, on the entire dataset. I would like to use radial basis function (RBF) kernel with parameter selection (there are two parameters for an RBF kernel: C and gamma). Usually, people select the hyperparameters of SVM using a development set, and then use the best hyperparameters based on the development set and apply it to the test set for evaluations. However, in my case, the original dataset is partitioned into 10 subsets. Sequentially one subset is tested using the classifier trained on the remaining 9 subsets. It is obviously that we do not have fixed training and test data. How should I do hyper-parameter estimation in this case?

Answer 1

You can do a "nested" cross-validation in one of two ways. First, you could do a k-fold within each training set. Or you could use one partition for development.

In the first case, you would use fold $i$ for testing and all other folds for training. Within each training set, you would do another k-fold cross-validation to optimize the parameters. So if fold $1$ is testing this time, then you would do another k-fold on folds $2$-$10$ to optimize the parameters.

In the second case, you would use fold $i$ for testing, fold $i+1$ for development, and all other folds for training. So if fold $1$ is testing this time, then fold $2$ would be development, and folds $3$-$10$ would be training. You would train models using folds $3$-$10$, find the parameters that perform best in fold $2$, and then use those parameters to make predictions in fold $1$. Then you iterate across all $i$ values.