RBF SVM use cases (vs logistic regression and random forest)

Question

Support Vector Machines with radial-base function kernel is a general-purpose supervised classifier.

While I know theoretical foundations for these SVMs, and their strong points, I am not aware of cases in which they are the preferred method. So, is there is a class of problems for which RBF SVMs are superior to other ML techniques? (Either in terms of score, or other - like robustness, easiness to start, interpretability etc.)

I am asking, as my default approach is centered around logistic regression (perhaps with some interactions), random forest and a bit of neural networks. None of my friends doing ML (some are Kaggle winners) is a SVM-user (but it may be an artifact of my community, or the problems they deal with).

Answer 1

I will try to answer this question with a combination of published evidence, personal experience, and speculation.

A) Published evidence.

The only paper I know that help answer the question is Delgado et al 2014 - Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? - JMLR which runs hundreds of different algorithms and implementations on 121 datasets fro UCI. They find that although RBF SVM are not the "best" algorithm (it is random forests if I remember correctly), it is among the top 3 (or 5).

If you consider that their selection of datasets is a "good sample" of real world problems, than SVM are definitively an algorithm that should be tried on new problems but one should try random forest first!

The limits on generalizing that result are that the datasets are almost all tall and skinny (n>>p), not very sparse - which I speculate should be more of a problem for RF, and not very big (both n and p).

Finally, still on the published evidence, I recommend two sites that compare different implementations of random forests:

• Benchmarking Random Forest Implementations

• Benchmarking Random Forest Classification

B) Personal experience.

I believe that papers such as Delgado et all very important for the machine learning community, so I tried to replicate their results under some different conditions. I ran some 15 different algorithms on 100+ binary datasets (from Delgado's set of datasets). I also think I was more careful on the selection of hyperparameters then they were.

My results is that the SVM was the "best algorithm" (mean rank 4.9). My take is that SVM passed RF because the original dataset contained many multiclass problems - which I will discuss in the speculation part - should be a problem for SVM.

EDIT (Jun/16):

But RF is way way faster, and it was the 2nd best algorithm (mean rank 5.6) followed by gbm (5.8), nnets (7.2), and so on). I did not try standard logistic regression in these problems, but I tried an elastic net (L1 and L2 regularized LR) but it did not perform well (mean rank 8.3)~

I have not yet finished analyzing the results or writing the paper so I cannot even point to a technical report with the results. Hopefully, in some weeks I can re-edit this answer and point to a technical report with the results.

The paper is available at http://arxiv.org/abs/1606.00930 It turns out that after the full analysis RF and SVM are almost equivalent in terms of expected error rate and SVM is fastest (to my surprise!!). I am no longer that emphatic in recommending RF (on speed grounds).

So my personal experience is that although SVM may get you some extra bit of accuracy, it is almost always a better choice to use a RF.

Also for larger problems, it may be impossible to use a batch SVM solver (I have never used a online SVM solver such as LASVM or others).

Finally I only used logistic regression in one situation. I was doing some "intense" feature engineering on a image classification problem (such as - combine or not two different descriptions of the image, and the dimensionality of the descriptions). And I used logistic regression to select among the many alternatives (because there is no hyperparameter search in LR). Once we settle in the best features (according to LR) we used a RF (selecting for the best hyperparameters) to get the final classifier.

C) Speculation

I have never seriously worked on multiclass problems, but my feeling is that SVM are not so good on them. The problem is not the issue between one-vs-one or one-vs-all solutions, but that all implementations that I know, will use the same hyperparameters for all (OVO or OVA) classifiers. Selecting the correct hyperparameters for SVM is so costly that none of the of-the-shelf implementations I know will do a search for each classifiers. I speculate that this is a problem for SVM (but not a problem for RF!!).

Then again, for multiclass problems I would go straight to RF.

Answer 2

I don't have sufficient privileges to be able to write comments, so I will just provide my input/observations here as an answer.

In my experience, Support Vector Classifiers (SVC) tend to be either at par or outperform the other methods when the binary classes are balanced. For unbalanced classes, SVC tends to perform poorly.

I don't often deal with multiclass problems, but I have seen some good results with SVC for multiclass problems as well.

Another thing I've noticed is that the curse of dimentionality doesn't seem to affect SVC as much as other modeling techniques. In other words, as I add more terms in the model, the other techniques start performing poorly on the test (or, holdout) set as compared to the training set. But not so much when I use SVC. Because of this reason, if model parsimony is not your priority, then SVC may be a better option as you can throw in a lot of terms without as much over-fitting as the other methods.

One of the issues I have with SVC is that it doesn't implicitly provide a measure (like predicted probability) to be able to rank order the observations. You could use Platt Scaling (implemented in sklearn.svm package in Python), but I have seen some inconsistencies. (I can share the details if anyone's interested.)

Not sure if this really answers your question, but these are my observations. Hope that helps.

Answer 3

RF and (RBF) SVM have different theories behind them, but assuming you have enough data, they perform similarly well. They both can learn complex functions and deal nicely with noisy and uninformative variables and outliers.

If you are trying to get best results for something like a kaggle, you would ensemble multiple models including RF and SVM anyway.

In non kaggle settings, you might consider how hard is it to implement the model, put it to production, make a prediction, interpret, explain it to a manager etc.

SVM (linear or highly regularized RBF) would be definitely preferred if you have small amount of data or you are dealing with a course of dimensionality. There is couple of reasons for it, one is that is better to look for maximum margin hyperplane instead of series of best splits on your features, also there is usually no need for a complex boundary because in high dimensional space there will be some hyperplane that can separate the data anyway. Another issue is that RF is harder to tune (has more parameters to tune), so you need more data.

Another think, cross validation can be very cheap and fast for SVM, especially LOOCV. Since only a few samples are support vectors (not always), you don have to retrain your classifier on every fold, but only when the data that are now in the test set were support vectors before. This can also make online learning easier.

Also, it might be cheaper to store support vectors than full trees.

Is often better to make probabilistic model than classifier. So, make model first and decision later. In that case logistic regression will be preferred. And you can still use kernels and regularization to make it behave like you want. Also, you will not use RF to answer questions like: correcting for age, lifestyle, sex and education, does drinking alcohol increase chance of dyeing of heart attack?

Some additional resource I found interesting: https://www.quora.com/What-are-the-advantages-of-different-classification-algorithms http://videolectures.net/solomon_caruana_wslmw/