Had a general questions. Are there any good non-linear models with regularization? I've heard of some linear models with regularization but not too many non-linear ones. I understand that you can use SVMs, which have built in regularization, but just curious what else might be available.



Most regularised linear models can be kernelised to give a non-linear equivalent. I made a MATLAB toolbox (sorry no manual yet) implementing the Generalised Kernel Machine, which is just a kernelisation of the generalised linear model. I rather like kernel logistic regression (implemented in the toolbox) as an alternative to the SVM as for many problems, you do actually want a probabilistic output. None of this is wildly novel, but I have found it useful, it has code for optimising the kernel and regularization parameters via approximate leave-one-out cross-validation). For details, see

G. C. Cawley, G. J. Janacek and N. L. C. Talbot, Generalised kernel machines, in Proceedings of the IEEE/INNS International Joint Conference on Neural Networks (IJCNN-2007), pages 1732-1737, Orlando, Florida, USA, August 12-17, 2007. (doi)

  • $\begingroup$ Thanks Dikran. I will definitely check this out. Out of curiosity, do you have any suggestions on modeling time series (e.g. dependent data) with SVMs? I've been trying different cross validation methods (rolling window type cross validation) but haven't had too much success. $\endgroup$ – tomas Feb 29 '12 at 14:14
  • $\begingroup$ No problem, I can't really advise on time-series as I don't have much experience with that. I seem to recall that Johan Suykens wrote a good paper on time series with the least-squares support vector machine (essentially kernelised ridge regression) for time series, IIRC in the late 90s/early 2000s, might be a good place to look (Johan's group has done lots of interesting things, so well worth checking out anyway!) $\endgroup$ – Dikran Marsupial Feb 29 '12 at 14:23
  • $\begingroup$ Thanks Dikran. He's definitely got some interesting papers. I will peruse! $\endgroup$ – tomas Feb 29 '12 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.