Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite
2

I am using R and e1071 package to tune a C-classification SVM.

My question is: regardless of the kernel type (linear, polynomial, radial basis or sigmoidal), is there any good criterion to choose the range in which cost and $\gamma$ parameters should range over and/or to choose what the granularity should be (that is, as an example, gamma = 10 ^ (1:2) or gamma = 1:2 or gamma = 100 ^ (1:2))?

I add a second question: can tune.svm() return the best kernel type, too?

Thanks,

share|improve this question

1 Answer

up vote 2 down vote accepted

Rather than using a grid search, it may be easier to use something like the Nelder-Mead simplex algorithm (which I believe has an R implementation) and use that to minimise the model selection criterion, and then you don't need to worry about the limits at all. Do minimise a continuous criterion though, for example the hinge loss on the test examples or radius-margin bound, rather than the test error rate (as that will be rather noisy and hard to optimised. Don't optimise the hyper-parameters too much though as it is also possible to get overfitting in model selection as well as in fitting the SVM.

share|improve this answer
Thank you for your suggestment, Dikran Marsupial. Let you have to decide which is the best kernel type and the best value of cost and $\gamma$ parameters: what would you optimize first, according to which criterion and why? – Lisa Ann Nov 21 '12 at 11:06
1  
Linear and RBF kernels are good reasonable choices to look at first, however the choice of kernel is very much data-dependent. The Nelder Mead method can optimise both parameters simultaneously, which is generally the best option. I rarely use SVMs these days, as Kernel Logistic Regression gives estimates of probability, which I find more useful than a binary decision for most real-world problems. Alternatively I use kernel ridge regression (AKA LS-SVM). The training procedure is much more straightforward, and you can get cross-validation almost for free, which I use for model selection. – Dikran Marsupial Nov 21 '12 at 12:27
Is the Kernel Logistic Regression like a non parametric Logit model (or similar to that one)? I will try it, thank you. Could you suggest me any R package with KLR? – Lisa Ann Nov 21 '12 at 12:33
KLR is essentially just a logistic regression model made non-linear using the same "kernel trick" used in the SVM. I am a MATLAB user, so I'm afraid I can't recommend a suitable R package. I've been meaning to learn R, so I may write one, one day! – Dikran Marsupial Nov 21 '12 at 13:21
Please, take a look at this: cran.r-project.org/web/packages/np/vignettes/np.pdf just from page 17 to page 19; is that similar to what you explained to me? – Lisa Ann Nov 21 '12 at 13:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.