Are neural networks better than SVMs?

Question

For some time now I have been studying both support vector machines and neural networks and I understand the logic behind each of these techniques. Very briefly described:

• In a support vector machine, using the kernel-trick, you "send" the data into a higher dimensional space where it can be linearly separable.

• In a neural network you perform a series of linear combinations mixed with (usually) non linear activation functions across several layers.

So far I have seen that neural networks tend to provide the best predictive results among machine learning alternatives. Of course, compared with other more classical tools like multivariate regression, they have some drawbacks, like providing little (if any) interpretability of the variables, while in regression the interpretability of the variables is immediate.

My question is: Neural networks seem to provide better predictive results than support vector machines, and both provide the same amount of interpretability (which is none). Is there any situation in which using a support vector machine would be better than using a neural network?

Answer 1

Short answer: On small data sets, SVM might be preferred.

Long answer:

Historically, neural networks are older than SVMs and SVMs were initially developed as a method of efficiently training the neural networks. So, when SVMs matured in 1990s, there was a reason why people switched from neural networks to SVMs. Later, as data sets grew larger and more complex, so that feature selection became a (even bigger) problem, while, at the same time, computational power rose, people switched back again.

This development already suggests that both have their strengths and weaknesses and that there is, as Haitao says, no free lunch.

Essentially, both methods do some kind of data transformation to "send" them into a higher dimensional space. What the kernel function does for the SVMs, the hidden layers do for neural networks. The last, output layer in the network also performs a linear separation of the so transformed data. So this is not the core difference.

To demonstrate this, I'm so free to use Haitao's example. As you can see below, a two-layer neural network, with 5 neurons in the hidden layer, can perfectly separate the two classes. The blue class can be fully enclosed in a pentagon (pale blue) area. Each neuron in the hidden layer determines a linear boundary---a side of the pentagon, producing, say, +1 when its input is a point on the "blue" side of the line and -1 otherwise (it could also produce 0, it doesn't really matter).

I have used different colours to highlight which neuron is responsible for which boundary. The output neuron (black) simply checks (performs a logical AND, which is again a linearly separable function) whether all hidden neurons give the same, "positive" answer. Observe that this last neuron has five inputs. I.e. its input is a 5-dimensional vector. So the hidden layers have transformed 2D data into 5D data.

Notice, however, that the boundaries drawn by the neural network are somewhat arbitrary. You can shift and rotate them slightly without really affecting the result. How the network draws the boundary is somewhat random; it depends on the initialisation of the weights and on the order you present the training set to it. This is where SVMs differ: They are guaranteed to draw the boundary mid-way between the closest points of the two classes! It can be (has been) shown that this boundary is the optimal one. Finding the boundary is a convex (quadratic) optimisation problem for which fast algorithms exist. Also, the kernel trick has the computational advantage that it's usually much faster to compute a single non-linear function than to pass the vector through many hidden layers.

However, since SVMs never compute the boundary explicitly, but through the weighted sum of the kernel functions over the pairs of the input data, the computational effort scales quadratically with the data set size. For large data sets this quickly becomes impractical.

Also, when the data are high-dimensional (think of images, with millions of pixels) the SVMs might become overwhelmed by the curse of dimensionality: It becomes too easy to draw a good boundary on the training set, but which has poor generalisation properties. Convolutional neural networks, on the other hand, are capable of learning the relevant features from the data.

There is also a computational power issue here: Today's networks like to use activation functions which are linear in segments, like the ReLUs, for a reason. Applying them is simple linear algebra, something GPUs are good at (because 3D graphics also involves a lot of matrix multiplications). So today's neural networks are, in part, a by-product of the gaming industry.

In summary, my suggestion is to use SVMs for low-dimensional, small data sets and neural networks for high-dimensional large data sets.

Answer 2

You may have heard of the "no free lunch theorem" in machine learning. For each model, there are pros and cons for specific data and use case.

So. NN is not better than SVM and I can give couple examples easily. One important argument is SVM is convex but NN is generally not. Having a convex problem is desirable because we have more tools to solve it more reliable.

If we know our data, we can pick a better model to fit data better. For example, if we have some data like donut shape. Like this

using SVM with right kernel is better than using NN and NN may overfit data in this case.

Answer 3

SVM is interesting if you have a kernel in mind that you know is appropriate, or a domain-specific kernel that would be difficult to express in a differentiable way (a common example might be a string-similarity space for DNA sequences). But what if you have no idea what kind of kernel you should use? What if your data is a wide collection of values and you're not even sure in advance which ones have relevance? You could spend human researcher time doing feature engineering, or you could try automatic kernel search methods, which are pretty expensive, but might even come up with something that could be considered interpretable, on a good day.

Or you could dump the whole thing into a DNN and train. What a neural net does through backprop and gradient descent could very well be considered to be learning a kernel, only instead of having a nice functional form, it's composed (literally) of a large number of applications of a basic nonlinearity, with some additions and multiplications thrown in. The next-to-last layer of a typical classification network is the result of this — it's a projection into a space with one dimension per neuron in that layer, where the categories are well-separated, and then the final result (ignoring the softmax, which is really just a kind of normalization) is an affine map of that space into one where the categories are axis-aligned, so the surfaces of separation come for free with the geometry (but we could send them backwards onto that second-to-last layer if we wanted).

The DNN classifier accomplishes something very similar to an SVM classifier, only it does it in a "dumb" way using gradient descent and repetition of simple differentiable units. But sometimes in computation, "dumb" has its advantages. Ease of application to GPUs (which love applying the same simple operation to a large number of data points in parallel) is one. The ability of SGD and minibatch gradient descent to scale up to very large numbers of examples with minimal loss of efficiency is another. Of course, it comes with its own downsides. If you make the wrong choices of NN architecture, initial weights, optimization method, learning rate, batch size, etc. then the stochastic training process might completely fail to converge, or take a million years to do so — whereas SVM training is basically deterministic.

(Forgive an amateur blundering around, oversimplifying, and abusing terminology; these are my personal experiences after 15 years or so of playing with this stuff on an occasional hobby level).

Answer 4

The paper "Every Model Learned by Gradient Descent Is Approximately a Kernel Machine" by Pedro Domingos, shows that every NN learned by gradient descent (not stochastic) is in essence a kernel machine. The kernel has rather a complicated form: $$ K(x, x^{'}) = \int_{c(t)} \nabla_w y(x) \nabla_w y(x^{'}) dt $$ Where $c(t)$ is the path traversed by a NN during the gradient descent. These is result is actually of a controversial practical importance, but gives rather an interesting interpretation and insights.

Answer 5

I am not an expert, so consider this as an opinion from someone who is still learning about these two fields.

Short Answer

Theoretically, No. DNNs can perform all the functions of SVMs and more.

Practically, mostly no. For most modern problems DNNs are a better choice. If your input data size is small and you are successful in finding a suitable kernel, however, an SVM may be a more efficient solution. But, if you can't determine a suitable kernel, NNs are then a better choice.

Long Answer

If we gloss over some details about the constraints of different implementations, deep neural networks are essentially universal function approximators. Given enough training time and a sufficiently robust architecture and training set (what we might call operating at the theoretical limit) they will essentially learn an approximation of an optimum solution. It might be an SVM-style kernel transformation or it might be some other function, it all depends on what approach actually works best for the task at hand.

In this sense, I think DNNs can be thought of as a super set that contains SVMs as one option among many. If an SVM is the most appropriate, the DNN will automatically learn the best kernel transformation for the task (assuming we are operating at the limit). So in a purely theoretical sense, I would say no, there is nothing an SVM can do that a DNN cannot (given enough layers of sufficient size, training, ect.)

From a practical perspective, however, which one is better comes down to the particular problem at hand and the constraints you are working with. In general though, I think DNNs are still often the better solution in practice, especially for most modern problems.

There are many ways of looking at this, but I think one of the more informative is thinking about the kernel. In some sense, SVMs are not "learning" a solution to your problem, they are just applying the kernel you pre-specified to your data. If that kernel results in your data being linearly separable in kernel space, great, it will work, if not, it will "fail" (to varying degrees). The key point being that you are not learning the appropriate kernel from your data, you are essentially guessing that your predetermined kernel will work. DNNs, on the other hand, can be thought of as simultaneously learning the appropriate kernel and applying it to your data. I think this is a major advantage of DNNs, especially in more "exploratory" cases.

If, however, you already "know the answer" to your classification problem (in the sense you know the input data has certain properties and have a kernel that is known to exploit those properties to make the classes linearly separable in kernel space) then an SVM may be a better solution. Given that the computational load scales quadratically with the data set size for SVMs, however, your data also needs to be small enough.

SVMs may also be preferred in that they are more "predictable"/theoretically founded. You know they are applying X kernel transformation to the data and attempting to linearly separate the classes by drawing a hyperplane equidistant between the two closest points from each class. That's a solution with defined behavior. What's going on inside of a very deep neural network, however, is much more difficult to ascertain.

In conclusion, from a theoretical perspective, I think DNNs provide power and flexibility beyond what SVMs can provide, and from a practical perspective, given that many modern problems involve fairly large input data sets and often no obviously appropriate kernel, I think NNs are often the more appropriate choice. The only benefits of SVMs over DNNs that I can see are that SVMs are more interpretable, easier to train, and and can be more computationally efficient for small datasets.

Answer 6

One specific benefit that these models have over SVMs is that their size is fixed: they are parametric models, while SVMs are non-parametric. That is, in an ANN you have a bunch of hidden layers with sizes h1 through hn depending on the number of features, plus bias parameters, and those make up your model. By contrast, an SVM (at least a kernelized one) consists of a set of support vectors, selected from the training set, with a weight for each. In the worst case, the number of support vectors is exactly the number of training samples (though that mainly occurs with small training sets or in degenerate cases) and in general its model size scales linearly. In natural language processing, SVM classifiers with tens of thousands of support vectors, each having hundreds of thousands of features, is not unheard of.

Also, online training of FF nets is very simple compared to online SVM fitting, and predicting can be quite a bit faster.