How to perform grid search effectively for tuning SVM parameters in cross validation?

Question

I have C and gamma parameters for RBF kernel to perform SVM classification through cross validation in R software. How to fix values for grid search to tune C and gamma parameters? For example I took grid ranging from [50 , 60 , 70 ....,600] for C and Gamma [ 0.05, 0.10,....,1]. I used a validation set for fine tuning the parameters. I fixed the gamma value and varied the C and got the optimum C value. Then I fixed the optimum C value and varied the gamma values to find the optimum gamma value. Is it right or are there any other way to perform effective grid search?

Answer 1

in R you can do this by using tune.svm function of e1071 package for eg

obj = tune.svm(x,y,cost=10:100,gamma=seq(0,3,0.1))

would give you best cost and gamma value

please note that the values for cost and gamma are for understanding purpose only

Answer 2

Both C and gamma are scale parameters, so the grid should be on a logarithmic scale. I find that doubling/halving C and gamma on adjacent grid points is a good compromise (still working on the paper). For most problems the validation error will be non-convex, but generally unimodal, so gradient free optimisation tools, such as the Nelder-Mead simplex method are fairly efficient (I suspect someone may have implemented that for R, but I am a MATLAB user).

Answer 3

The C and gamma parameters influence each other, as you can see here:

_{(source: scikit-learn.org)}

The performance of your solution depends on the initial fixed gamma value. If you choose a bad initial gamma value, you'll end up with a bad solution.

The easiest, but most time consuming way to find C and gamma is to test the whole grid of C x gamma values.

I often use some kind of (bayesian) optimization algorithm like this one (it's for Python, but similar should exist for R). It normally finds good C and gamma values in relatively few iterations.

PS.: the C and gamma values should be taken from a logarithmic grid of values like 10**[-5..5] - using a linear grid like [50, 60, ... ,600] won't work well.