Many authors of papers I read affirm SVMs is superior technique to face their regression/classification problem, aware that they couldn't get similar results through NNs. Often the comparison states that

SVMs, instead of NNs,

  • Have a strong founding theory
  • Reach the global optimum due to quadratic programming
  • Have no issue for choosing a proper number of parameters
  • Are less prone to overfitting
  • Needs less memory to store the predictive model
  • Yield more readable results and a geometrical interpretation

Is it seriously a broadly accepted thought? Don't quote No-Free Lunch Theorem or similar statements, my question is about practical usage of those techniques.

On the other side, which kind of abstract problem you definitely would face with NN?

  • 3
    I think the question might be a bit broad. But in practice NNs seem to be a lot more tunable with choice of NN structure, whereas SVMs have fewer parameters. There's two questions, if an NN were optimally set up for solving a problem how would it fare vs SVM? And in the hands of the average practioner, how does SVM compare with NN? – Patrick Caldon Jun 8 '12 at 3:10
  • 2
    @PatrickCaldon I understand your point of view, but more parameters to deal with do not always mean better tool, if you do not know how to configure them in a suitable way. Even if possibile, a long study might be needed; or, you might not need so broad tunability for the purpose of your applciation – stackovergio Jun 8 '12 at 4:54
  • 1
    that's my point. Which question how does the tool work in ideal circumstances on particular problems? or how does the tool work for most people most of the time? I think the biggest component here is the person btw. Because of this I think the relevant factors are often: How hard is each tool to learn? Are there experts around who know how to use it? etc. That can explan a lot of "I got good performance out of X" – Patrick Caldon Jun 8 '12 at 7:30
  • 1
    From what I know, multilayer feedforward ANN are universal approximators more or less irrespective of the activation function. I am not aware of a similar result for SVM which dependto my knowledge much more on the kernel function used. – Momo Jun 8 '12 at 13:10
  • 2
    In practical usage, I find NNs a bit more practical due to training time. Non-Linear SVMs just can't handle large N very well. Both algorithms can overfit and both need strong regularization. – Shea Parkes Jun 8 '12 at 19:42
up vote 36 down vote accepted

It is a matter of trade-offs. SVMs are in right now, NNs used to be in. You'll find a rising number of papers that claim Random Forests, Probabilistic Graphic Models or Nonparametric Bayesian methods are in. Someone should publish a forecasting model in the Annals of Improbable Research on what models will be considered hip.

Having said that for many famously difficult supervised problems the best performing single models are some type of NN, some type of SVMs or a problem specific stochastic gradient descent method implemented using signal processing methods.


Pros of NN:

  • They are extremely flexible in the types of data they can support. NNs do a decent job at learning the important features from basically any data structure, without having to manually derive features.
  • NN still benefit from feature engineering, e.g. you should have an area feature if you have a length and width. The model will perform better for the same computational effort.

  • Most of supervised machine learning requires you to have your data structured in a observations by features matrix, with the labels as a vector of length observations. This restriction is not necessary with NN. There is fantastic work with structured SVM, but it is unlikely it will ever be as flexible as NNs.


Pros of SVM:

  • Fewer hyperparameters. Generally SVMs require less grid-searching to get a reasonably accurate model. SVM with a RBF kernel usually performs quite well.

  • Global optimum guaranteed.


Cons of NN and SVM:

  • For most purposes they are both black boxes. There is some research on interpreting SVMs, but I doubt it will ever be as intuitive as GLMs. This is a serious problem in some problem domains.
  • If you're going to accept a black box then you can usually squeeze out quite a bit more accuracy by bagging/stacking/boosting many many models with different trade-offs.

    • Random forests are attractive because they can produce out-of-bag predictions(leave-one-out predictions) with no extra effort, they are very interpretable, they have an good bias-variance trade-off(great for bagging models) and they are relatively robust to selection bias. Stupidly simple to write a parallel implementation of.

    • Probabilistic graphical models are attractive because they can incorporate domain-specific-knowledge directly into the model and are interpretable in this regard.

    • Nonparametric(or really extremely parametric) Bayesian methods are attractive because they produce confidence intervals directly. They perform very well on small sample sizes and very well on large sample sizes. Stupidly simple to write a linear algebra implementation of.

  • 31
    With the recent rise of deep learning, NNs can be considere "more in" than SVMs, I'd say. – bayerj Jun 9 '12 at 18:03

The answer to your question is in my experience "no", SVMs are not definitely superior, and which works best depends on the nature of the dataset at hand and on the relative skill of the operator with each set of tools. In general SVMs are good because the training algorithm is efficient, and it has a regularisation parameter, which forces you to think about regularisation and over-fitting. However, there are datasets where MLPs give much better performance than SVMs (as they are allowed to decide their own internal representation, rather than having it pre-specified by the kernel function). A good implementation of MLPs (e.g. NETLAB) and regularisation or early stopping or architecture selection (or better still all three) can often give very good results and be reproducible (at least in terms of performance).

Model selection is the major problem with SVMs, choosing the kernel and optimising the kernel and regularisation parameters can often lead to severe over-fitting if you over-optimise the model selection criterion. While the theory under-pinning the SVM is a comfort, most of it only applies for a fixed kernel, so as soon as you try to optimise the kernel parameters it no longer applies (for instance the optimisation problem to be solved in tuning the kernel is generally non-convex and may well have local minima).

  • 6
    I fully agree with this. I am currently training SVMs and ANNs on brain-computer interface data and there are some data sets where SVMs are better and some data sets where ANNs are better. The interesting thing is: when I average the performance over all the data sets I am using, SVMs and ANNs reach exactly the same performance. Of course, this is not a proof. It's just an anecdote. :) – alfa Jun 30 '12 at 18:32

I will just try to explain my opinion that appeared to be shared by most of my friends. I have the following concerns about NN that are not about SVM at all:

  1. In a classic NN, the amount of parameters is enormously high. Let's say you have the vectors of the length 100 you want to classify into two classes. One hidden layer of the same size as an input layer will lead you to more then 100000 free parameters. Just imagine how badly you can overfit (how easy is it to fall to local minimum in such a space), and how many training points you will need to prevent that (and how much time will you need to train then).
  2. Usually you have to be a real expert to chose the topology at a glance. That means that if you want to get good results you should perform lots of experiments. That's why it's easier to use SVM and tell, that you couldn't get similar results with NN.
  3. Usually NN results are not reproducible. Even if you run your NN training twice, you will probably get different results due to the randomness of a learning algorithm.
  4. Usually you have no interpretation of the results at all. That is a small concern, but anyway.

That doesn't mean that you should not use NN, you should just use it carefully. For example, Convolutional NN can be extremely good for image processing, other Deep NN proved to be good for other problems as well.

Hope it will help.

  • 1
    To make ANN results reproducible, seed the random function. – Franck Dernoncourt Dec 27 '16 at 18:21
  • @Franck That's not real reproducibility. – sanity Mar 16 '17 at 3:14

I am using neural networks for most problem. The point is that it's in most cases more about the experience of the user than about the model. Here are some reasons why I like NNs.

  1. They are flexible. I can throw whatever loss I want at them: hinge loss, squared, cross entropy, you name it. As long as it is differentiable, I can even design a loss which fits my needs exactly.
  2. They can be treated probabilistically: Bayesian neural networks, variational Bayes, MLE/MAP, everything is there. (But in some cases more difficult.)
  3. They are fast. Most MLPs will be two matrix multiplications and one nonlinearity applied component wise in between. Beat that with an SVM.

I will go through your other points step by step.

Have a strong founding theory

I'd say, NNs are equally strong in that case: since you train them in a probabilistic framework. That makes the use of priors and a Bayesian treatment (e.g. with variational techniques or approximations) possible.

Reach the global optimum due to quadratic programming

For one set of hyperparameters. However, the search for good hps is non-convex, and you won't know whether you found the global optimum as well.

Have no issue for choosing a proper number of parameters

With SVMs, you have to select hyper parameters as well.

Needs less memory to store the predictive model

You need to store the support vectors. SVMs will not in general be cheaper to store MLPs, it depends on the case.

Yield more readable results and a geometrical interpretation

The top layer of an MLP is a logistic regression in the case of classification. Thus, there is a geometrical interpretation (separating hyper plane) and a probabilistic interpretation as well.

  • Why do I need to store support vectors? Isn't it enough to store the hyperplane/maring of SVM? – Julian Jun 30 '17 at 8:15
  • That's because the hyper plane is represented through support vectors. To calculate the distance of a new point from it, you will iteratore over those. – bayerj Jun 19 at 18:57

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.