Neural networks vs support vector machines: are the second definitely superior?

Question

Many authors of papers I read affirm SVMs is superior technique to face their regression/classification problem, aware that they couldn't get similar results through NNs. Often the comparison states that

SVMs, instead of NNs,

• Have a strong founding theory
• Reach the global optimum due to quadratic programming
• Have no issue for choosing a proper number of parameters
• Are less prone to overfitting
• Needs less memory to store the predictive model
• Yield more readable results and a geometrical interpretation

Is it seriously a broadly accepted thought? Don't quote No-Free Lunch Theorem or similar statements, my question is about practical usage of those techniques.

On the other side, which kind of abstract problem you definitely would face with NN?

Answer 1

It is a matter of trade-offs. SVMs are in right now, NNs used to be in. You'll find a rising number of papers that claim Random Forests, Probabilistic Graphic Models or Nonparametric Bayesian methods are in. Someone should publish a forecasting model in the Annals of Improbable Research on what models will be considered hip.

Having said that for many famously difficult supervised problems the best performing single models are some type of NN, some type of SVMs or a problem specific stochastic gradient descent method implemented using signal processing methods.

---

Pros of NN:

• They are extremely flexible in the types of data they can support. NNs do a decent job at learning the important features from basically any data structure, without having to manually derive features.

• NN still benefit from feature engineering, e.g. you should have an area feature if you have a length and width. The model will perform better for the same computational effort.

• Most of supervised machine learning requires you to have your data structured in a observations by features matrix, with the labels as a vector of length observations. This restriction is not necessary with NN. There is fantastic work with structured SVM, but it is unlikely it will ever be as flexible as NNs.

---

Pros of SVM:

• Fewer hyperparameters. Generally SVMs require less grid-searching to get a reasonably accurate model. SVM with a RBF kernel usually performs quite well.

• Global optimum guaranteed.

---

Cons of NN and SVM:

• For most purposes they are both black boxes. There is some research on interpreting SVMs, but I doubt it will ever be as intuitive as GLMs. This is a serious problem in some problem domains.

• If you're going to accept a black box then you can usually squeeze out quite a bit more accuracy by bagging/stacking/boosting many many models with different trade-offs.

  • Random forests are attractive because they can produce out-of-bag predictions(leave-one-out predictions) with no extra effort, they are very interpretable, they have an good bias-variance trade-off(great for bagging models) and they are relatively robust to selection bias. Stupidly simple to write a parallel implementation of.

  • Probabilistic graphical models are attractive because they can incorporate domain-specific-knowledge directly into the model and are interpretable in this regard.

  • Nonparametric(or really extremely parametric) Bayesian methods are attractive because they produce confidence intervals directly. They perform very well on small sample sizes and very well on large sample sizes. Stupidly simple to write a linear algebra implementation of.

Answer 2

The answer to your question is in my experience "no", SVMs are not definitely superior, and which works best depends on the nature of the dataset at hand and on the relative skill of the operator with each set of tools. In general SVMs are good because the training algorithm is efficient, and it has a regularisation parameter, which forces you to think about regularisation and over-fitting. However, there are datasets where MLPs give much better performance than SVMs (as they are allowed to decide their own internal representation, rather than having it pre-specified by the kernel function). A good implementation of MLPs (e.g. NETLAB) and regularisation or early stopping or architecture selection (or better still all three) can often give very good results and be reproducible (at least in terms of performance).

Model selection is the major problem with SVMs, choosing the kernel and optimising the kernel and regularisation parameters can often lead to severe over-fitting if you over-optimise the model selection criterion. While the theory under-pinning the SVM is a comfort, most of it only applies for a fixed kernel, so as soon as you try to optimise the kernel parameters it no longer applies (for instance the optimisation problem to be solved in tuning the kernel is generally non-convex and may well have local minima).

Answer 3

I will just try to explain my opinion that appeared to be shared by most of my friends. I have the following concerns about NN that are not about SVM at all:

1. In a classic NN, the amount of parameters is enormously high. Let's say you have the vectors of the length 100 you want to classify into two classes. One hidden layer of the same size as an input layer will lead you to more then 100000 free parameters. Just imagine how badly you can overfit (how easy is it to fall to local minimum in such a space), and how many training points you will need to prevent that (and how much time will you need to train then).
2. Usually you have to be a real expert to chose the topology at a glance. That means that if you want to get good results you should perform lots of experiments. That's why it's easier to use SVM and tell, that you couldn't get similar results with NN.
3. Usually NN results are not reproducible. Even if you run your NN training twice, you will probably get different results due to the randomness of a learning algorithm.
4. Usually you have no interpretation of the results at all. That is a small concern, but anyway.

That doesn't mean that you should not use NN, you should just use it carefully. For example, Convolutional NN can be extremely good for image processing, other Deep NN proved to be good for other problems as well.

Hope it will help.

Answer 4

I am using neural networks for most problem. The point is that it's in most cases more about the experience of the user than about the model. Here are some reasons why I like NNs.

1. They are flexible. I can throw whatever loss I want at them: hinge loss, squared, cross entropy, you name it. As long as it is differentiable, I can even design a loss which fits my needs exactly.
2. They can be treated probabilistically: Bayesian neural networks, variational Bayes, MLE/MAP, everything is there. (But in some cases more difficult.)
3. They are fast. Most MLPs will be two matrix multiplications and one nonlinearity applied component wise in between. Beat that with an SVM.

I will go through your other points step by step.

Have a strong founding theory

I'd say, NNs are equally strong in that case: since you train them in a probabilistic framework. That makes the use of priors and a Bayesian treatment (e.g. with variational techniques or approximations) possible.

Reach the global optimum due to quadratic programming

For one set of hyperparameters. However, the search for good hps is non-convex, and you won't know whether you found the global optimum as well.

Have no issue for choosing a proper number of parameters

With SVMs, you have to select hyper parameters as well.

Needs less memory to store the predictive model

You need to store the support vectors. SVMs will not in general be cheaper to store MLPs, it depends on the case.

Yield more readable results and a geometrical interpretation

The top layer of an MLP is a logistic regression in the case of classification. Thus, there is a geometrical interpretation (separating hyper plane) and a probabilistic interpretation as well.

Answer 5

In some ways these two broad categories of machine learning techniques are related. Though not perfect, two papers I have found helpful in showing the similarities in these techniques are below

Ronan Collobert and Samy Bengio. 2004. Links between perceptrons, MLPs and SVMs. In Proceedings of the twenty-first international conference on Machine learning (ICML '04). ACM, New York, NY, USA, 23-. DOI: https://doi.org/10.1145/1015330.1015415

and

Andras, Peter. (2002). The Equivalence of Support Vector Machine and Regularization Neural Networks. Neural Processing Letters. 15. 97-104. 10.1023/A:1015292818897.