4
$\begingroup$

I am currently reading the recent papers mainly written by Y. Bengio [1],[2],[3].

There are very strong claims about poorness of Kernel methods on recognizing handwritings in many general cases but there are no references for the claims of this poorness. I want to know if this is really the case in Machine Learning research. e.g.: " Unfortunately, devising such similarity measures, even for a problem as basic as digit recognition, has proved difficult, despite almost 50 years of active research. Furthermore, if such a good task-specific kernel were finally designed, it may be inapplicable to other classes of problems."

Improve this question
$\endgroup$
5
  • $\begingroup$ Support vector machines are among the top performing approaches on the MNIST data set (handwritten digit recognition). I haven't read the papers you refer to, but those claims seem to require some context. $\endgroup$
    – Marc Claesen
    Jan 9, 2014 at 8:35
  • $\begingroup$ @MarcClaesen, well apparently right now the best algorithms are for deep learning methods on MNIST data. You can refer to: 1) ieeexplore.ieee.org/xpl/… 2)ieeexplore.ieee.org/xpl/… $\endgroup$
    – Cupitor
    Jan 9, 2014 at 8:42
  • $\begingroup$ I never said SVMs are the best, but they are at the top which shows that the "poorness of kernel methods" should at the very least be framed in some context. Kernel methods are also very popular in computer vision applications like object recognition, which would not be the case if they were as bad as the OP suggests. In some settings kernel methods will surely perform poorer than others, which is why I say some context is necessary. In general, kernel methods are not bad at all for visual pattern recognition. $\endgroup$
    – Marc Claesen
    Jan 9, 2014 at 9:03
  • $\begingroup$ @MarcClaesen, well according to these papers, kernels in kernel methods are highly tailored for specific application which is not the case for these deep learning methods. Moreover they claim poorness is very general specially in handwriting recognition because of highly entangled and high curvature manifolds for digits. I suggest you to take a look at: ieeexplore.ieee.org/xpl/… $\endgroup$
    – Cupitor
    Jan 9, 2014 at 9:12
  • $\begingroup$ This is just a piece of propaganda from someone who is advocating deep learning instead of kernel methods. It should not be taken as impartial. $\endgroup$
    – seanv507
    Jun 1, 2015 at 9:38

2 Answers 2

Reset to default
2
$\begingroup$

Without further context, such a claim seems a bit too extreme to me. Standard SVMs are no deep architectures, they are flat architectures which perform a clever pattern comparison.

What I mean is that the decision function for SVMs is, $$ f(x) = \text{sign}\left[\sum_{n}\alpha_{n}K(x_{n},x)-b\right] $$ so you can see it as a voting, where $K(x_{i},x)$ is a measure of the similarity, and $\alpha_{i}$ is the weight of each vote. Still, these "simple" algorithm has achieved great performance in many tasks in machine vision and natural language processing.

What is the difference? Convolutional neural networks, for example, can be thought as built of two blocks: a first one which learns a good representation of the input data in the form of features which are invariant to scaling, rotations and so on, and a second layer which learns to classify the objects based on those features. The SVMs do not learn any features. They compare samples.

Following this idea there is the paper "Large-scale Learning with SVM and Convolutional Nets for Generic Object Categorization", where the features learned by a convolutional network are used to train a SVM, achieving great results. Better than the convolutional network on that task, which speaks for the ability of the SVM as a discriminative classifier.

Another issue are structured SVMs (in the setting of structured learning) where they are competitive which deep networks (which are also able to solve such tasks).

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Excellent answer. I'd add that I think the general trend in the computer vision community is to use convolutional nets (at least) as inputs, but there was a time way back in 2006 when spatial pyramid matching kernels showed promise. On one-hand, you're hard-coding a feature extraction procedure that a neural networks are able to learn. On the other, it's more robust wrt hyperparameter selection. See: www-cvr.ai.uiuc.edu/ponce_grp/publication/paper/cvpr06b.pdf $\endgroup$
    – Jessica Collins
    Aug 12, 2014 at 0:14
  • $\begingroup$ @jpmuc Very nice answer! I would like to add that one can also view svm's as a linear model on top of extracted features $\phi(x)$. However, in contrast to neural nets, the transformation in svm's is fixed (the user must specify the the kernel) whereas in neural nets the transformation is learnable. $\endgroup$
    – ado sar
    Sep 28, 2023 at 16:05
0
$\begingroup$

Kernels can be extended to nearly any kind of data, but the modelling has to be done very carefully. And this is the reason, why many people are scared of it. Its difficult to handle non-understood methods. Its just not enough to apply some gaussian kernels and to perform grid search on the scaling.

This critisism is not just related to kernel methods, but to all somewhat mathematical approaches in computer vision, machine learning etc.

If you read also "older" [1] papers, you will indeed see that the kernels methods are used for more than 50 years everywhere, where computing is required. So it cannot be that bad. :) However, after a quick look on your links, I could see that Y. Bengio is also working a lot on deep learning, which is also a kind of kernel based regression.

I wouldn't try to find a "go-for-all" method, but to select a specific problem and a good-looking paper and rebuild it from scratch in order to get good feeling for problems and benefits of a method. On your way you will have to develop techniques to "debug" the kernel regression/classification/neural networks, which will give you new ideas and force you to solve "unexplained" phenomena.

Good luck!

Improve this answer
$\endgroup$
5
  • $\begingroup$ Vote up! Thank you for your detailed writing. However there are points that I have hard time agreeing with you: - That was exactly why I asked this question.... That if it was so bad why poeple are still using it... on the other hand....there was the example of old days in artificial intelligence for 20 years and then a sudden stop. So I cannot say that which ones is the case really for Kernels. - I am really curious to know how do you categorise Deep learning in kernel based regression?? -The truth is their suggestion in 2007 paper together with Lecun is actually a go-for-all goal... $\endgroup$
    – Cupitor
    Dec 9, 2013 at 23:10
  • 2
    $\begingroup$ Hi, well in my opinion nothing disappeared. I think that the names just changed. Kernels were called "spline models in reproducing hilbert space" and the Gaussian kernel was known as the "green function" developed to solve finite-elements problems numerically. Also neural networks are called "deep learning" today. And they definetely didnt stop to exist. When it came to public "dislike" of the NN, researcher started to call the matrix multiplication $Wx$ "linear embedding" instead of a neural network. :) $\endgroup$
    – mojovski
    Dec 10, 2013 at 9:22
  • 1
    $\begingroup$ And as for the categorization of DL and Kernel based regression, a learner can always be seen as a regressor with class labels as required output. The "activation functions" are then your kernels. $\endgroup$
    – mojovski
    Dec 10, 2013 at 9:23
  • $\begingroup$ Hi and thank you very much for this update. Then about your last point. I agree that you can make a kernel out of it. The point is with a normal kernel approach as Bengio claims(not me :D) it seems that one usually uses local kernels and if you want to go for complex kernels then you probably need a layer over it...right? $\endgroup$
    – Cupitor
    Dec 10, 2013 at 14:03
  • $\begingroup$ As an additional point, kernel methods are highly dependant on their linear regression applied after selecting the feature functions, however the layers in a neural network can add a remarkable non-linearity to the model. Am I right on this too? $\endgroup$
    – Cupitor
    Dec 10, 2013 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.