17
$\begingroup$

More of a general question. I'm running an rbf SVM for predictive modeling. I think my current program definitely needs a bit of a speed up. I use scikit learn with a coarse to fine grid search + cross validation.

Each SVM run takes around a minute, but with all the iterations, I'm still finding it too slow. Assuming I eventually multi thread the cross validation part over multiple cores, any recommendations on speeding up my program? Any faster implementations of SVMs? I've heard of some GPU SVMs, but haven't digged into it much. Any users and is it faster?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ SGDClassifier in scikit-learn is very fast, but for linear SVMs. You might ask the scikit-learn guys, also add tag scikit-learn. $\endgroup$
    – denis
    Commented Apr 19, 2012 at 14:56
  • $\begingroup$ Non-linear kernel SVM are doomed to be slow. Perhaps you should start playing with linear models (check out Vowpal Wabbit) and then go to non-linear. You can even get some non-linearity by creating more complicated features with linear models. Quite often non-linear models result in some incremental performance increase at big computational expense. Nothing against non-linear kernel SVM but just to keep in mind, from practical point of view. $\endgroup$
    – Vladislavs Dovgalecs
    Commented Mar 26, 2015 at 4:57
  • 1
    $\begingroup$ You can speed up by using specialized tuning libraries for hyperparameter search, which are way more efficient than grid search (ie. require testing far less sets of hyperparameters). Examples of tuning libraries include Optunity and Hyperopt. $\endgroup$
    – Marc Claesen
    Commented Mar 26, 2015 at 5:57

6 Answers 6

Reset to default
17
$\begingroup$

Google's Sofia algorithm contains an extremely fast implementation of a linear SVM. It's one of the fastest SVMs out there, but I think it only supports classification, and only supports linear SVMs.

There's even an R package!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Your link now says "package ‘RSofia’ was removed from the CRAN repository." Any idea why? $\endgroup$
    – James Hirschorn
    Commented Nov 2, 2016 at 5:33
  • $\begingroup$ @JamesHirschorn The developer probably stopped maintaining it. You can install it from the CRAN archive. $\endgroup$
    – Zach
    Commented Nov 2, 2016 at 16:33
10
$\begingroup$

The easiest speedup you're going to get is running the cross-validation in parallel. Personally, I like the caret package in R, which uses foreach as a backend. It makes it very easy to farm the cross-validation and grid search out to multiple cores or multiple machines.

Caret can handle many different models, including rbf SVMs:

library(caret)
library(doMC)
registerDoMC()
model <-  train(Species ~ ., data = iris, method="svmRadial", 
    trControl=trainControl(method='cv', number=10))
> confusionMatrix(model)
Cross-Validated (10 fold) Confusion Matrix 

(entries are percentages of table totals)

            Reference
Prediction   setosa versicolor virginica
  setosa       32.4        0.0       0.0
  versicolor    0.0       30.9       2.0
  virginica     0.9        2.4      31.3

Note that the doMC() library is only available on mac and linux, it should be run from the command line, not from a GUI, and it breaks any models from RWeka. It's also easy to use MPI or SNOW clusters as parallel backend, which don't have these issues.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks Zach. I believe scikits also has a way of running cross validation in parallel which is what I plan to do. Excluding that, any other suggestions on speedup? Thanks. $\endgroup$
    – tomas
    Commented Feb 17, 2012 at 14:35
  • $\begingroup$ @danjeharry: parallel cross-validation is really the low hanging fruit here, and I strongly suggest you explore that first. Beyond that, I don't know a lot about speeding up SVMs specifically. If you can find a parallel SVM algorithm, that might be a good idea too. How many rows/columns is the data set you're using to train? $\endgroup$
    – Zach
    Commented Feb 17, 2012 at 14:55
  • $\begingroup$ Thanks Zach I will look into parallel cv. I'm doing about 650 attributes and 5000 examples. $\endgroup$
    – tomas
    Commented Feb 21, 2012 at 21:06
4
$\begingroup$

I realize this is a quite old question, but it's also possible (depending on the size of your dataset it can be more or less effective) to use low-dimensional approximations of the kernel feature map and then use that in a linear-SVM. See http://scikit-learn.org/stable/modules/kernel_approximation.html

Improve this answer
$\endgroup$
2
$\begingroup$

Have a look at Python's multiprocessing module. It makes parallelizing things really easy and is perfect for cross validation.

Improve this answer
$\endgroup$
2
$\begingroup$

R has a great GPU-accelerated svm package rpusvm, it takes ~20 seconds to train on 20K samples*100 dimensions, and I found that the CPU is never overloaded by it, so it uses the GPU efficiently. However, it requires a NVIDIA GPU.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It also requires purchasing an rpudplus license. $\endgroup$
    – Avraham
    Commented Apr 14, 2022 at 17:37
1
$\begingroup$

Alert: This is a shameless plug.

Consider DynaML a Scala based ML library I am working on. I have implemented Kernel based LS-SVM (Least Squares Support Vector Machines) along with automated Kernel tuning, using grid search or Coupled Simulated Annealing.

http://mandar2812.github.io/DynaML/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.