hyper parameter optimization grid search issues I keep running into the same problem while doing a grid search to optimize the C and gamma parameters of an SVC.  Every time i do the grid search, the best values seem to occur at around C = 100000 and gamma = 0.0000001 or even more extreme numbers. I am using 10 fold cross validation on my test set to obtain these values. My question is:  Should I be focusing on a specific range for C and gamma, in order for my model to generalize to new data better, or should i just trust the grid search and use the extreme values found.  I was under the impression that using a c value this extreme would guarantee over fitting.  I got this image from the sci kit website regarding hyper parameter optimization.  They say that the best results are at C = 1 and gamma = 0.1.  Is there something wrong with the results in the bottom left corner?  As it seems there is 3 squares with the same validation accuracy down there.   Thanks in advance.

 A: Some thoughts:


*

*Don't use accuracy. It's virtually guaranteed to pick the wrong model. Look at ROC AUC, Brier score or another proper scoring rule. Reference: "The Case Against Accuracy" Foster Provost, Tom Fawcett, Ron Kohavi.

*It is generally true that the best hyperparameters are poorly identified, and there will tend to be some sort of correspondence between $\gamma,C$. Smaller sample sizes tend to exacerbate this.

*Applying the 1 standard error rule can help hedge against the over-fitting that would occur by fixing hyperparameters at the maximum.

*Grid search is a slow, unintelligent way to go about tuning hyperparameters, in the sense that you spend lots of time exploring areas that are plausibly terrible, like the upper-left region of your plot. Bayesian Optimizaton and Particle Swarm Optimization are both faster, more intelligent ways to search the hyperparameter space.

*Standard ranges for $C$ and $\gamma$ are $C\in [2^{-5}, 2^{15}]$ and $\gamma\in[2^{-15}, 2^{3}], $ but there's nothing sacrosanct about those values, except the observation that larger $C$ tends towards overfitting.
