Define ranges for nested cross validation in SVM parameter tuning

Question

I have a question regarding parameter tuning in nested cross validation. I have already found some excellent questions and comments on the topic (Training with the full dataset after cross-validation?). However, they do not really address my question.

Suppose I want to do model checking for a SVM classification with a k-fold outer cross validation. For each fold of the outer cross validation I use a nested cross validation for parameter tuning (cost, gamma) of the respective SVM using grid-search. My question is how and where to define the parameter ranges for the grid-search then.

The options I see:

Option 1: Define a range once and use this same range to tune the method in each fold. If this is a valid approach, how would you set the parameter values ex-ante if you have no idea how they perform?

Option 2: Define an individual range in each fold and refine the search manually several times. Also, what would be a good criterion to stop refining the grid-search in each fold? This would also imply that you would need to do the tuning manually in each fold.

Any thoughts on what is a good approach here?

Answer 1

Assuming you're sticking with grid search (there are many other options, such as Random Search, Particle Swarm Optimization, CMA-ES, Simulated Annealing, etc).

Option 1: Define a range once and use this same range to tune the method in each fold. If this is a valid approach, how would you set the parameter values ex-ante if you have no idea how they perform?

This is the most common approach and quite robust to overfitting (with a coarse grid) when compared to more complex options. The a priori choice though is subject to heuristics. If you have the computational resources to investigate a huge parameter space, try to space well the evaluation points. Usually, both $C$ and $\gamma$ are searched in logarithmic scale.

Option 2: Define an individual range in each fold and refine the search manually several times. Also, what would be a good criterion to stop refining the grid-search in each fold? This would also imply that you would need to do the tuning manually in each fold.

This is much more laborious and quite prone to overfitting, but would save you from the issue in grid search when the optimal parameters fall out of your grid. A good stop criterion depends on your objective function, but you could define an a priori target objective, or stop when no improvement higher than a threshold is made between iterations.

---

Overall, if sticking to grid search, Option 1 sounds better because it's reproducible.