2
$\begingroup$

Just a general question. Are there any good non-linear SVM (kernelized) implementations that include a regularization component (e.g. $L_1$, SCAD etc)? I've been looking around but man there are a lot of different bits and pieces of code in various languages everywhere. I've seen some stuff on R, but it looks like most regularized SVM implementations use a linear kernel.

Any suggestions thoughts welcome. Thanks all!

$\endgroup$
3
$\begingroup$

It is perhaps worth pointing out that an SVM with an Lq or other regularisation term acting on the Lagrange multipliers (the alphas) is not really any longer a kernel model, as the interpretation of a linear model constructed in a kernel induced feature space is almost certainly lost (and along with it the generalisation bounds on which the SVM is based). This doesn't mean that it won't work better than any other sort of SVM, just that some of the justification for kernel approaches is lost, so there may not be a good reason for concentrating on SVMs and something like an import vector machine might be more appropriate.

If a sparse model non-linear is what is required, you may also just use LASSO/LARS/Elastic net, after pre-transforming the inputs using a radial (or some other) basis functions etc.

$\endgroup$
  • $\begingroup$ Thanks Dikran I was thinking about the same thing. So I could basically take my inputs, map it to a higher dimensional space with one of the kernel functions, and use the new feature space to run a linear regularized model? Then basically I would just need to cross-validate (or whatever method) to determine the kernel parameter (likely to use rbf) and the regularization parameter? $\endgroup$ – tomas Mar 2 '12 at 19:31
  • $\begingroup$ Hey sorry for the additional comment, but the import vector machine seems like a really interesting idea. Out of curiosity, are there any implementations of it? $\endgroup$ – tomas Mar 3 '12 at 0:01
  • $\begingroup$ Yes, that is the basic idea, although it doesn't give you a true kernel method is the regularisation term becomes dependent on the sample of data on which you contruct the model. Cross-validation is probably going to be the best approach. Sadly, I don't know of an implementation of the import vector machine. I was quite interested in sparse approximation to kernel machines a while back, but I haven't tried the IVM (yet). $\endgroup$ – Dikran Marsupial Mar 3 '12 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.