12
$\begingroup$

I read about SVMs and learnt that they are solving an optimization problem and max margin idea was very reasonable.

Now, using kernels they can find even non-linear separation boundaries which was great.

So far, I really do not have any idea how SVMs (a special kernel machine) and kernel machines are related to neural networks?

Consider the comments by Yann Lecun => here:

kernel methods were a form of glorified template matching

and here too:

For example, some people were dazzled by kernel methods because of the cute math that goes with it. But, as I’ve said in the past, in the end, kernel machines are shallow networks that perform “glorified template matching”. There is nothing wrong with that (SVM is a great method), but it has dire limitations that we should all be aware of.

So my questions are:

  1. How is SVM related to neural network? How is it a shallow network?
  2. SVM solves an optimization problem with a well defined objective function, how is it doing template matching? What is the template here to which an input is matched?

I guess these comments need a thorough understanding of high dimensional spaces, neural nets and kernel machines but so far I have been trying and couldn't grasp the logic behind it. But it is surely interesting to note the connections between two very very different ml techniques.

EDIT: I think understanding SVMs from a Neural perspective would be great. I am looking for a thorough mathematics backed answer to the above two questions, so as to really understand the link between SVMs and Neural Nets, both in the case of linear SVM and SVMs with the kernel trick.

Improve this question
$\endgroup$
2
  • $\begingroup$ SVMs are fairly easy & fast to train given an appropriate kernel. Some tasks don't need deep neural net. $\endgroup$
    – Vladislavs Dovgalecs
    Feb 23, 2017 at 17:23
  • $\begingroup$ @xeon hi, can you take a look at the answer, I suppose it needs improvement. thanks. $\endgroup$
    – Rafael
    Mar 3, 2017 at 7:32

1 Answer 1

Reset to default
7
$\begingroup$
  1. How is SVM related to neural network? How is it a shallow network?

The SVM is a single layer neural network with the hinge loss as loss function and exclusively linear activation. The concept has been alluded in previous threads, such as this one: Single layer NeuralNetwork with RelU activation equal to SVM?

  1. SVM solves an optimization problem with a well defined objective function, how is it doing template matching? What is the template here to which an input is matched?

The Gram Matrix (Kernel Matrix, if you prefer) is a measure of similarity. As the SVM allows sparse solutions, prediction becomes a matter of comparing your sample with the templates, i.e. the support vectors.

Improve this answer
$\endgroup$
4
  • $\begingroup$ thanks for the answer, please explain a bit more with some maths preferably. That would be really great :) $\endgroup$
    – Rafael
    Feb 24, 2017 at 5:45
  • $\begingroup$ I more or less understand the template matching thing, but I didn't get the statement: As SVM allows sparse solutions.. what has sparse solutions got to do anything here? Prediction by definition is done by a weighing similarity with the templates, so I don't get where sparsity comes from. Also, please add a few lines regarding the hinge loss activation function. Thanks a lot :) $\endgroup$
    – Rafael
    Mar 1, 2017 at 10:09
  • $\begingroup$ How about Every Model Learned by Gradient Descent Is Approximately a Kernel Machine? Deep networks were also doing template matching? $\endgroup$
    – n0p
    Apr 9, 2021 at 12:56
  • $\begingroup$ @nop Of course, each decision is quite explicitly a pairwise comparison with all training data (encoded in the weights of a network). They are, of course, approximately kernel machines. But Kernel Machines are explicit about it, so it's easier to not dilute the term. $\endgroup$
    – Firebug
    Apr 9, 2021 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.