I have some data (total N = 100,000 rows). I randomly selected 10% from it to become the validation set to help identify the best set of hyperparameters. To do that I am conducting grid search based on 5-fold cross-validation.

I understand in scikit-learn, I can simply obtain the best parameter set as an output. But I want to repeat the 5-fold cross-validation grid search for 3 times.

The grid search options are as follows for SVC:

C:  0.001,  0.01,   0.1
tol:    0.001,  0.01,   0.1
class_weight:   None,   balanced    

Say, after repeating it 3 times, the followings are the best parameter setting for each iteration:

                         Accuracy   C        tol    class_weight
Best in iteration 1          0.8    0.1     0.01    None
Best in iteration 2         0.75    0.001   0.001   balanced
Best in iteration 3         0.78    0.001   0.001   None

What is the best way to select the final (most optimal) parameter set? Do I simply choose Iteration #1 since its accuracy is better than the other two?

Or because their accuracy are fairly similar, can I make a case to choose 0.001 for tol since it appears more often than 0.01? If to be based on highest frequency for each parameter, then I will be choosing C=0.001, tol=0.001, and class_weight=None. Or would basing it on frequency be a bad idea as parameters relate and influence each other for the algorithm to work optimally?

  • 1
    $\begingroup$ I think most optimal is superfluous as optimal is already the best there is. $\endgroup$ – Richard Hardy Sep 25 '19 at 20:31
  • $\begingroup$ Well, it's just semantics. Each iteration is supposed to generate an optimal parameter set, I want to distinguish the final chosen set derived from these optimal sets, so it's arbitrary called "most optimal". $\endgroup$ – KubiK888 Sep 25 '19 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.