# About SVM cost and gamma parameters tuning

I am using R and e1071 package to tune a C-classification SVM.

My question is: regardless of the kernel type (linear, polynomial, radial basis or sigmoidal), is there any good criterion to choose the range in which cost and $\gamma$ parameters should range over and/or to choose what the granularity should be (that is, as an example, gamma = 10 ^ (1:2) or gamma = 1:2 or gamma = 100 ^ (1:2))?

I add a second question: can tune.svm() return the best kernel type, too?

Thanks,

-

Rather than using a grid search, it may be easier to use something like the Nelder-Mead simplex algorithm (which I believe has an R implementation) and use that to minimise the model selection criterion, and then you don't need to worry about the limits at all. Do minimise a continuous criterion though, for example the hinge loss on the test examples or radius-margin bound, rather than the test error rate (as that will be rather noisy and hard to optimised. Don't optimise the hyper-parameters too much though as it is also possible to get overfitting in model selection as well as in fitting the SVM.

-
Thank you for your suggestment, Dikran Marsupial. Let you have to decide which is the best kernel type and the best value of cost and $\gamma$ parameters: what would you optimize first, according to which criterion and why? –  Lisa Ann Nov 21 '12 at 11:06
Linear and RBF kernels are good reasonable choices to look at first, however the choice of kernel is very much data-dependent. The Nelder Mead method can optimise both parameters simultaneously, which is generally the best option. I rarely use SVMs these days, as Kernel Logistic Regression gives estimates of probability, which I find more useful than a binary decision for most real-world problems. Alternatively I use kernel ridge regression (AKA LS-SVM). The training procedure is much more straightforward, and you can get cross-validation almost for free, which I use for model selection. –  Dikran Marsupial Nov 21 '12 at 12:27
Is the Kernel Logistic Regression like a non parametric Logit model (or similar to that one)? I will try it, thank you. Could you suggest me any R package with KLR? –  Lisa Ann Nov 21 '12 at 12:33
KLR is essentially just a logistic regression model made non-linear using the same "kernel trick" used in the SVM. I am a MATLAB user, so I'm afraid I can't recommend a suitable R package. I've been meaning to learn R, so I may write one, one day! –  Dikran Marsupial Nov 21 '12 at 13:21
Please, take a look at this: cran.r-project.org/web/packages/np/vignettes/np.pdf just from page 17 to page 19; is that similar to what you explained to me? –  Lisa Ann Nov 21 '12 at 13:39