I was reading this question about selecting hyper-parameters for a support vector machine classifier, where grid-search is presented as one option. Which one is correct, either
for f in folds for c in c_grid for s in sigma_grid # build svm # find best (c, s) pair (considering single-fold). # find best (c, s) pair (considering all-folds)
for c in c_grid for s in sigma_grid for f in folds # build svm # find best (c, s)
So does one repeat cross-validation for each (c, s) pair or does one select optimal (c, s) pair within each iteration of cross-validation?
If the first option is correct, how does one select the optimal (c,s) pair? The values could be different for different folds.
(Assume that this is the inner-loop of nested cross-validation, as per the other question.)