Poorness of Kernel methods on visual pattern recegnition? I am currently reading the recent papers mainly written by Y. Bengio [1],[2],[3].
There are very strong claims about poorness of Kernel methods on recognizing handwritings in many general cases but there are no references for the claims of this poorness. I want to know if this is really the case in Machine Learning research. e.g.:
" Unfortunately, devising such similarity measures, even for a problem as basic as digit recognition, has proved difficult, despite almost 50 years of active research. Furthermore, if such a good task-specific kernel were finally designed, it may be inapplicable to other classes of problems."
 A: Without further context, such a claim seems a bit too extreme to me. Standard SVMs are no deep architectures, they are flat architectures which perform a clever pattern comparison.
What I mean is that the decision function for SVMs is,
$$
f(x) = \text{sign}\left[\sum_{n}\alpha_{n}K(x_{n},x)-b\right]
$$
so you can see it as a voting, where $K(x_{i},x)$ is a measure of the similarity, and $\alpha_{i}$ is the weight of each vote. Still, these "simple" algorithm has achieved great performance in many tasks in machine vision and natural language processing.
What is the difference? Convolutional neural networks, for example, can be thought as built of two blocks: a first one which learns a good representation of the input data in the form of features which are invariant to scaling, rotations and so on, and a second layer which learns to classify the objects based on those features. The SVMs do not learn any features. They compare samples.
Following this idea there is the paper "Large-scale Learning with SVM and Convolutional Nets for Generic Object Categorization", where the features learned by a convolutional network are used to train a SVM, achieving great results. Better than the convolutional network on that task, which speaks for the ability of the SVM as a discriminative classifier.
Another issue are structured SVMs (in the setting of structured learning) where they are competitive which deep networks (which are also able to solve such tasks). 
A: Kernels can be extended to nearly any kind of data, but the modelling has to be done very carefully. And this is the reason, why many people are scared of it.
Its difficult to handle non-understood methods. Its just not enough to apply some gaussian kernels and to perform grid search on the scaling.
This critisism is not just related to kernel methods, but to all somewhat mathematical approaches in computer vision, machine learning etc.
If you read also "older" [1] papers, you will indeed see that the kernels methods are used for more than 50 years everywhere, where computing is required. So it cannot be that bad. :)
However, after a quick look on your links, I could see that Y. Bengio is also working a lot on deep learning, which is also a kind of kernel based regression.
I wouldn't try to find a "go-for-all" method, but to select a specific problem and a good-looking paper and rebuild it from scratch in order to get good feeling for problems and benefits of a method. 
On your way you will have to develop techniques to "debug" the kernel regression/classification/neural networks, which will give you new ideas and force you to solve "unexplained" phenomena.
Good luck!
