Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm using libsvm and I noticed that everytime I call svmtrain(), I create a new model and that there seems to be no option to put data in an existing model. Is this possible to do however? Am I just not seeing this aspect in libsvm?

share|improve this question
    
I'm not sure what you mean by 'put data in an existing model'? Can you give an example of some other technique (not SVM) that allows you to do this? Take for example logistic regression; if you add new data you will get a new set of co-efficients re-trained on the existing set without reference to which data is 'new' or 'old', it's all just training data. I guess if you are using a gradient descent type solver you could save time by initialising at the previously optimised values, which will probably be close to the new solution. Is this what you mean? –  Bogdanovist Jun 20 '12 at 23:07
add comment

3 Answers 3

up vote 10 down vote accepted

It sounds like you're looking for an "incremental" or "online" learning algorithm. These algorithms let you update a classifier with new examples, without retraining the entire thing from scratch.

It's definitely possible with support vector machines, though I believe libSVM doesn't presently support it. It might be worth taking a look at several other packages that do offer it, including

PS: @Bogdanovist: There's a pretty extensive literature on this. kNN is obviously and trivially incremental. One could turn (some) bayesian classifiers into incremental classifiers by storing counts instead of probabilities. STAGGER, AQ* and some (but not all) of the ID* family of decision tree algorithms are also incremental, off the top of my head.

share|improve this answer
1  
Interesting, thanks for the heads up. I had seen the term 'online' bandied about before, but hadn't realised the technical significance (I thought it literally meant 'can haz internetz'). –  Bogdanovist Jun 21 '12 at 0:18
    
thanks I'll take a look! –  mugetsu Jun 21 '12 at 15:42
    
Glad to help! I should have mentioned this above, but some online/incremental algorithms actually do give more weight to the "newest" examples, which may or may not be useful, depending on your application (e.g., great for predicting twitter topics, less awesome for cancer research). –  Matt Krause Jun 21 '12 at 17:51
add comment

Most of the online/incremental SVM utilities are for linear kernels and I suppose its not as difficult as it is for non-linear kernels.

Some of the notable Online/incremental SVM tools currently available:
+ Leon Bottous's LaSVM: It supports both linear and non-linear kernels. C++ code
+ Bordes's LaRank: It supports both linear and non-linear kernels. C++ code . It seems the link is broken now :-(
+ Gert Cauwenberghs' code incremental and decremental: supports both linear and nonlinear kernels. Matlab code .
+ Chris Diehl's Incremental SVM Learning: supports both linear and non-linear kernels. Matlab code.
+ Alistair Shilton's SVMHeavy: Only Binary classification and regression. C++ code
+ Francesco Parrella's OnlineSVR: Only Regression. Matlab and C++.
+ Pegasos: Both linear and nonlinear. C and Matlab code. A java interface.
+ Langford's Vowpal Wabbit: Not sure :-(
+ Koby Crammer’s MCSVM: Both linear and non-linear. C code

A more updated list can be found on my Quora answer.

share|improve this answer
    
(+1) Welcome to the site. That is quite an exhaustive list! :) –  cardinal Mar 12 '13 at 12:48
add comment

Another possibility is alpha-seeding. I am not aware whether libSVM supports it. The idea is to divide a huge amount of training data into chunks. Then you train a SVM on the first chunk. As the resulting support vectors are nothing but some of the samples of your data, you take those and use them to train your SVM with the next chunk. Also, you use that SVM to compute a initial estimate of the alpha values for the next iteration (seeding). Therefore, the benefits are twofold: each of the problems is smaller and through smart initialization they converge even faster. This way you simplify a huge problem into sequentially solving a series of simpler steps.

share|improve this answer
    
is there any library out there applying this method? –  d.putto Mar 12 '13 at 15:52
    
apparently libsvm already does it, or at least some variant of the algorithm work.caltech.edu/~htlin/program/libsvm –  juampa Mar 12 '13 at 19:37
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.