How do I use Cross-validation to evaluate/validate different settings (“hyperparameters”) in machine learning?

Question

I've gone through a few posts about Cross-validation such as post1, post2, specially the scikit-learn doc, which says

When evaluating different settings (“hyperparameters”) for estimators, such as the C setting that must be manually set for an SVM, there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally.

I am aware that CV could help solve the problem of lacking of data by using $k-1$ of the folds as training data and the remaining part as test data.

However, I don't see how does Cross-validation help to solve the problem of evaluating different settings.

Consider this code

>>> clf = svm.SVC(kernel='linear', C=1)
>>> scores = cross_val_score(clf, X, y, cv=5)
>>> scores
array([0.96..., 1.  ..., 0.96..., 0.96..., 1.        ])

which is just one possible setting C=1, how about others, e.g. C=0.5

How do I use CV to evaluate/validate different settings?

>>> clf = svm.SVC(kernel='linear', C=0.5)
>>> scores = cross_val_score(clf, X, y, cv=5)

---

Note: I understand how does CV work within one setting, I would just like to know how it help to evaluate/validate different settings. Could someone please give a hint? Thanks in advance.

Assume k to refers to the number of folds and I am comparing 10 different settings, from C=0.1 to C=1.0. Each setting produces its own scores like array([0.96..., 1. ..., 0.96..., 0.96..., 1. ]) with 'cv=5'. So, k is equal to 5, n is equal to 10, I would pick the best from all 10 averages, right?

Answer 1

Normally, using some kind of hyper-parameter search, e.g. grid search, for each of the hyper-parameters, you'll train your model on $n-1$ folds, and test on the holdout fold, repeat this experiment $n$ times and typically report the average of the scores you have over these $n$ holdout folds. This way, you'll quantify how good you've done for each of the different hyper-parameter configurations and pick the best one to apply and report your final success on the test set.