I'm using libsvm and I noticed that everytime I call svmtrain(), I create a new model and that there seems to be no option to put data in an existing model. Is this possible to do however? Am I just not seeing this aspect in libsvm?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ I'm not sure what you mean by 'put data in an existing model'? Can you give an example of some other technique (not SVM) that allows you to do this? Take for example logistic regression; if you add new data you will get a new set of co-efficients re-trained on the existing set without reference to which data is 'new' or 'old', it's all just training data. I guess if you are using a gradient descent type solver you could save time by initialising at the previously optimised values, which will probably be close to the new solution. Is this what you mean? $\endgroup$– BogdanovistJun 20, 2012 at 23:07
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
2
It sounds like you're looking for an "incremental" or "online" learning algorithm. These algorithms let you update a classifier with new examples, without retraining the entire thing from scratch.
It's definitely possible with support vector machines, though I believe libSVM doesn't presently support it. It might be worth taking a look at several other packages that do offer it, including
- Gert Cauwenbergh's 2000 NIPS paper (with code) http://www.isn.ucsd.edu/svm/incremental/
- Pegasos (which is available by itself or as part of dlib)
- SVM Heavy http://people.eng.unimelb.edu.au/shiltona/svm/
PS: @Bogdanovist: There's a pretty extensive literature on this. kNN is obviously and trivially incremental. One could turn (some) bayesian classifiers into incremental classifiers by storing counts instead of probabilities. STAGGER, AQ* and some (but not all) of the ID* family of decision tree algorithms are also incremental, off the top of my head.
answered Jun 21, 2012 at 0:09
-
1$\begingroup$ Interesting, thanks for the heads up. I had seen the term 'online' bandied about before, but hadn't realised the technical significance (I thought it literally meant 'can haz internetz'). $\endgroup$ Jun 21, 2012 at 0:18
-
$\begingroup$ Glad to help! I should have mentioned this above, but some online/incremental algorithms actually do give more weight to the "newest" examples, which may or may not be useful, depending on your application (e.g., great for predicting twitter topics, less awesome for cancer research). $\endgroup$ Jun 21, 2012 at 17:51
$\begingroup$
$\endgroup$
1
Most of the online/incremental SVM utilities are for linear kernels and I suppose its not as difficult as it is for non-linear kernels.
Some of the notable Online/incremental SVM tools currently available:
+ Leon Bottous's LaSVM: It supports both linear and non-linear kernels. C++ code
+ Bordes's LaRank: It supports both linear and non-linear kernels. C++ code . It seems the link is broken now :-(
+ Gert Cauwenberghs' code incremental and decremental: supports both linear and nonlinear kernels. Matlab code .
+ Chris Diehl's Incremental SVM Learning: supports both linear and non-linear kernels. Matlab code.
+ Alistair Shilton's SVMHeavy: Only Binary classification and regression. C++ code
+ Francesco Parrella's OnlineSVR: Only Regression. Matlab and C++.
+ Pegasos: Both linear and nonlinear. C and Matlab code. A java interface.
+ Langford's Vowpal Wabbit: Not sure :-(
+ Koby Crammer’s MCSVM: Both linear and non-linear. C code
A more updated list can be found on my Quora answer.
-
$\begingroup$ (+1) Welcome to the site. That is quite an exhaustive list! :) $\endgroup$– cardinalMar 12, 2013 at 12:48
$\begingroup$
$\endgroup$
2
Another possibility is alpha-seeding. I am not aware whether libSVM supports it. The idea is to divide a huge amount of training data into chunks. Then you train a SVM on the first chunk. As the resulting support vectors are nothing but some of the samples of your data, you take those and use them to train your SVM with the next chunk. Also, you use that SVM to compute a initial estimate of the alpha values for the next iteration (seeding). Therefore, the benefits are twofold: each of the problems is smaller and through smart initialization they converge even faster. This way you simplify a huge problem into sequentially solving a series of simpler steps.
-
$\begingroup$ is there any library out there applying this method? $\endgroup$– d.puttoMar 12, 2013 at 15:52
-
$\begingroup$ apparently libsvm already does it, or at least some variant of the algorithm work.caltech.edu/~htlin/program/libsvm $\endgroup$– jpmucMar 12, 2013 at 19:37
$\begingroup$
$\endgroup$
Another option if you are seeking an "incremental" solution can be found here...
An extension of LIBLINEAR which allows for incremental learning.
