16
$\begingroup$

More of a general question. I'm running an rbf SVM for predictive modeling. I think my current program definitely needs a bit of a speed up. I use scikit learn with a coarse to fine grid search + cross validation.

Each SVM run takes around a minute, but with all the iterations, I'm still finding it too slow. Assuming I eventually multi thread the cross validation part over multiple cores, any recommendations on speeding up my program? Any faster implementations of SVMs? I've heard of some GPU SVMs, but haven't digged into it much. Any users and is it faster?

$\endgroup$
  • 1
    $\begingroup$ SGDClassifier in scikit-learn is very fast, but for linear SVMs. You might ask the scikit-learn guys, also add tag scikit-learn. $\endgroup$ – denis Apr 19 '12 at 14:56
  • $\begingroup$ Non-linear kernel SVM are doomed to be slow. Perhaps you should start playing with linear models (check out Vowpal Wabbit) and then go to non-linear. You can even get some non-linearity by creating more complicated features with linear models. Quite often non-linear models result in some incremental performance increase at big computational expense. Nothing against non-linear kernel SVM but just to keep in mind, from practical point of view. $\endgroup$ – Vladislavs Dovgalecs Mar 26 '15 at 4:57
  • 1
    $\begingroup$ You can speed up by using specialized tuning libraries for hyperparameter search, which are way more efficient than grid search (ie. require testing far less sets of hyperparameters). Examples of tuning libraries include Optunity and Hyperopt. $\endgroup$ – Marc Claesen Mar 26 '15 at 5:57
16
$\begingroup$

Google's Sofia algorithm contains an extremely fast implementation of a linear SVM. It's one of the fastest SVMs out there, but I think it only supports classification, and only supports linear SVMs.

There's even an R package!

$\endgroup$
  • $\begingroup$ Your link now says "package ‘RSofia’ was removed from the CRAN repository." Any idea why? $\endgroup$ – James Hirschorn Nov 2 '16 at 5:33
  • $\begingroup$ @JamesHirschorn The developer probably stopped maintaining it. You can install it from the CRAN archive. $\endgroup$ – Zach Nov 2 '16 at 16:33
10
$\begingroup$

The easiest speedup you're going to get is running the cross-validation in parallel. Personally, I like the caret package in R, which uses foreach as a backend. It makes it very easy to farm the cross-validation and grid search out to multiple cores or multiple machines.

Caret can handle many different models, including rbf SVMs:

library(caret)
library(doMC)
registerDoMC()
model <-  train(Species ~ ., data = iris, method="svmRadial", 
    trControl=trainControl(method='cv', number=10))
> confusionMatrix(model)
Cross-Validated (10 fold) Confusion Matrix 

(entries are percentages of table totals)

            Reference
Prediction   setosa versicolor virginica
  setosa       32.4        0.0       0.0
  versicolor    0.0       30.9       2.0
  virginica     0.9        2.4      31.3

Note that the doMC() library is only available on mac and linux, it should be run from the command line, not from a GUI, and it breaks any models from RWeka. It's also easy to use MPI or SNOW clusters as parallel backend, which don't have these issues.

$\endgroup$
  • $\begingroup$ Thanks Zach. I believe scikits also has a way of running cross validation in parallel which is what I plan to do. Excluding that, any other suggestions on speedup? Thanks. $\endgroup$ – tomas Feb 17 '12 at 14:35
  • $\begingroup$ @danjeharry: parallel cross-validation is really the low hanging fruit here, and I strongly suggest you explore that first. Beyond that, I don't know a lot about speeding up SVMs specifically. If you can find a parallel SVM algorithm, that might be a good idea too. How many rows/columns is the data set you're using to train? $\endgroup$ – Zach Feb 17 '12 at 14:55
  • $\begingroup$ Thanks Zach I will look into parallel cv. I'm doing about 650 attributes and 5000 examples. $\endgroup$ – tomas Feb 21 '12 at 21:06
4
$\begingroup$

I realize this is a quite old question, but it's also possible (depending on the size of your dataset it can be more or less effective) to use low-dimensional approximations of the kernel feature map and then use that in a linear-SVM. See http://scikit-learn.org/stable/modules/kernel_approximation.html

$\endgroup$
2
$\begingroup$

Have a look at Python's multiprocessing module. It makes parallelizing things really easy and is perfect for cross validation.

$\endgroup$
2
$\begingroup$

R has a great GPU-accelerated svm package rpusvm, it takes ~20 seconds to train on 20K samples*100 dimensions, and I found that the CPU is never overloaded by it, so it uses the GPU efficiently. However, it requires a NVIDIA GPU.

$\endgroup$
1
$\begingroup$

Alert: This is a shameless plug.

Consider DynaML a Scala based ML library I am working on. I have implemented Kernel based LS-SVM (Least Squares Support Vector Machines) along with automated Kernel tuning, using grid search or Coupled Simulated Annealing.

http://mandar2812.github.io/DynaML/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.