# How to select the most optimal hyperparameter in grid-search cross validation if the process is repeated X (i.e. 3) times?

I have some data (total N = 100,000 rows). I randomly selected 10% from it to become the validation set to help identify the best set of hyperparameters. To do that I am conducting grid search based on 5-fold cross-validation.

I understand in scikit-learn, I can simply obtain the best parameter set as an output. But I want to repeat the 5-fold cross-validation grid search for 3 times.

The grid search options are as follows for SVC:

C:  0.001,  0.01,   0.1
tol:    0.001,  0.01,   0.1
class_weight:   None,   balanced


Say, after repeating it 3 times, the followings are the best parameter setting for each iteration:

                         Accuracy   C        tol    class_weight
Best in iteration 1          0.8    0.1     0.01    None
Best in iteration 2         0.75    0.001   0.001   balanced
Best in iteration 3         0.78    0.001   0.001   None


What is the best way to select the final (most optimal) parameter set? Do I simply choose Iteration #1 since its accuracy is better than the other two?

Or because their accuracy are fairly similar, can I make a case to choose 0.001 for tol since it appears more often than 0.01? If to be based on highest frequency for each parameter, then I will be choosing C=0.001, tol=0.001, and class_weight=None. Or would basing it on frequency be a bad idea as parameters relate and influence each other for the algorithm to work optimally?

• I think most optimal is superfluous as optimal is already the best there is. – Richard Hardy Sep 25 '19 at 20:31
• Well, it's just semantics. Each iteration is supposed to generate an optimal parameter set, I want to distinguish the final chosen set derived from these optimal sets, so it's arbitrary called "most optimal". – KubiK888 Sep 25 '19 at 21:24