Is it possible to append training data to existing SVM models?

Question

I'm using libsvm and I noticed that everytime I call svmtrain(), I create a new model and that there seems to be no option to put data in an existing model. Is this possible to do however? Am I just not seeing this aspect in libsvm?

Answer 1

It sounds like you're looking for an "incremental" or "online" learning algorithm. These algorithms let you update a classifier with new examples, without retraining the entire thing from scratch.

It's definitely possible with support vector machines, though I believe libSVM doesn't presently support it. It might be worth taking a look at several other packages that do offer it, including

• Gert Cauwenbergh's 2000 NIPS paper (with code) http://www.isn.ucsd.edu/svm/incremental/
• Pegasos (which is available by itself or as part of dlib)
• SVM Heavy http://people.eng.unimelb.edu.au/shiltona/svm/

PS: @Bogdanovist: There's a pretty extensive literature on this. kNN is obviously and trivially incremental. One could turn (some) bayesian classifiers into incremental classifiers by storing counts instead of probabilities. STAGGER, AQ* and some (but not all) of the ID* family of decision tree algorithms are also incremental, off the top of my head.

Answer 2

Most of the online/incremental SVM utilities are for linear kernels and I suppose its not as difficult as it is for non-linear kernels.

Some of the notable Online/incremental SVM tools currently available:
+ Leon Bottous's LaSVM: It supports both linear and non-linear kernels. C++ code
+ Bordes's LaRank: It supports both linear and non-linear kernels. C++ code . It seems the link is broken now :-(
+ Gert Cauwenberghs' code incremental and decremental: supports both linear and nonlinear kernels. Matlab code .
+ Chris Diehl's Incremental SVM Learning: supports both linear and non-linear kernels. Matlab code.
+ Alistair Shilton's SVMHeavy: Only Binary classification and regression. C++ code
+ Francesco Parrella's OnlineSVR: Only Regression. Matlab and C++.
+ Pegasos: Both linear and nonlinear. C and Matlab code. A java interface.
+ Langford's Vowpal Wabbit: Not sure :-(
+ Koby Crammer’s MCSVM: Both linear and non-linear. C code

A more updated list can be found on my Quora answer.

Answer 3

Another possibility is alpha-seeding. I am not aware whether libSVM supports it. The idea is to divide a huge amount of training data into chunks. Then you train a SVM on the first chunk. As the resulting support vectors are nothing but some of the samples of your data, you take those and use them to train your SVM with the next chunk. Also, you use that SVM to compute a initial estimate of the alpha values for the next iteration (seeding). Therefore, the benefits are twofold: each of the problems is smaller and through smart initialization they converge even faster. This way you simplify a huge problem into sequentially solving a series of simpler steps.