Were SVMs developed as a method of efficiently training neural networks?

Question

This answer contains the following statement:

SVMs were initially developed as a method of efficiently training the neural networks.

Is that correct? Could anyone provide a reference?

Answer 1

Let me start with a quote by a co-inventor of the Support Vector Machines, Isabelle Guyon:

At the time, everybody was working on multi-layer Perceptrons (the ancestors of deep learning), and my first work on optimal margin algorithms was simply some active set method to enlarge their margin, inspired by the 'minover.' Bernhard Boser, my husband, was even making dedicated hardware for MLPs! I had heated discussions with Vladimir Vapnik, who shared my office and was pushing another optimal margin algorithm that he invented in the 1960's.

The invention of SVMs happened when Bernhard decided to implement Vladimir's algorithm in the three months we had left before we moved to Berkeley.

Or, somewhat more "official":

SVM is a learning technique developed by V. Vapnik and his team (AT&T Bell Labs., 1985) that can be seen as a new method for training polynomial, neural network, or Radial Basis Functions classifiers.

[E. Osuna, R. Freund, and F. Girosit: "Training support vector machines: an application to face detection". Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 130-136, 1997]

But the whole history is, of course, much longer and more convoluted. Below I offer a very short and slim timeline, just as a brief overview:

• in 1958, Rosenblatt, presents the first biologically motivated artificial neural network [F. Rosenblatt: "The perceptron: a probabilistic model for infomation storage and organization in the Brain". Psychological Review, 65(6):386-408]

• in 1964, Vapnik and Chervonenkis present the maximum margin criterion for training the perceptron [V. Vapnik and A. Chervonenkis: "A note on one class of perceptrons". Automation and Remote Control, 25(1):112-120]

• also in 1964, Aizerman, Braverman, and Roznoer introduce the interpretation of kernels ("potential functions") as inner products in a feature space and prove that "perceptron can be considered to be a realization of potential function method" [M.A. Aizerman, E.M. Braverman and L.I. Roznoer: "Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning". Automation and Remote Control, 25(12):821-837]

• in 1969, Minsky and Papert publish their book "Perceptrons: An Introduction to Computational Geometry" and show the limitations of (single-layer) perceptrons.

• in 1986, Rumelhart, Hinton, and Williams (re)invent the backpropagation algorithm and make learning in multi-layer perceptrons ("feed-forward neural networks") possible [D.E. Rumelhart, G.E. Hinton, and R.J. Williams: "Learning representations by back-propagating errors". Nature 323(6088):533–536].

• in 1992, Boser, Guyon, and Vapnik (the three from the introductory quote) present what we now call the "Support Vector Machine". They state: "The technique is applicable to a wide variety of classification functions, including Perceptrons [...]" [B.E. Boser, I.M. Guyon, and V.N. Vapnik: "A Training Algorithm for Optimal Margin Classifiers". Proceedings of the 5th Annual Workshop on Computational Learning Theory (COLT'92), 144-152]