I have some data (total N = 100,000 rows). I randomly selected 10% from it to become the validation set to help identify the best set of hyperparameters. To do that I am conducting grid search based on 5-fold cross-validation.
I understand in
scikit-learn, I can simply obtain the best parameter set as an output. But I want to repeat the 5-fold cross-validation grid search for 3 times.
The grid search options are as follows for
C: 0.001, 0.01, 0.1 tol: 0.001, 0.01, 0.1 class_weight: None, balanced
Say, after repeating it 3 times, the followings are the best parameter setting for each iteration:
Accuracy C tol class_weight Best in iteration 1 0.8 0.1 0.01 None Best in iteration 2 0.75 0.001 0.001 balanced Best in iteration 3 0.78 0.001 0.001 None
What is the best way to select the final (most optimal) parameter set? Do I simply choose Iteration #1 since its accuracy is better than the other two?
Or because their accuracy are fairly similar, can I make a case to choose 0.001 for
tol since it appears more often than 0.01? If to be based on highest frequency for each parameter, then I will be choosing
C=0.001, tol=0.001, and class_weight=None. Or would basing it on frequency be a bad idea as parameters relate and influence each other for the algorithm to work optimally?