Let’s take for example a binary classification task
To me it seems that both Deep Learning and Kernel based Methods seem to share the idea of not trying to learn a complex classification function directly in the Input Space but to first perform a mapping into a more convenient (in order to solve the task i.e. to optimize the error function) representation and then to solve the task in that space, learning a simpler function
It seems to me that Deep Learning is mostly about performing an effective Representation Learning.
One of the main conjectures about Deep Learning is its capability to develop a Hierarchical Multi-Layer representation, moving from a High Dimensional Input Space (e.g. the Image Space) to more Abstract Representations into Lower Dimensional Spaces.
In Supervised Learning, the Representation(s) Learned is (are) biased to allow for a (easy, to some extent) solution of a certain task and indeed final layers are typically classical ones (e.g. Logistic Regression on top of a CNN Stack for Object Classification)
So basically this idea looks similar to the Kernel based Strategies one: instead of solving a problem directly in the Input Space looking for a very complex function (i.e. searching over a huge hypothesis space) it’s better to investigate the possibility of developing a projection into a space where the solution can be easier (i.e. reducing the hypothesis space): so instead of learning a complex nonlinear classification function in the input space it’s better to develop a mapping into an intermediate abstract space (or a set of intermediate representations) where the solution in the final layer representation can be a linear classifier.
One of the major differences I see is that in Kernel based Strategies, like SVM, the Kernel is a prior typically defined as a result of some domain expertise, while in Deep Learning based Strategies, like CNN, the Convolutional Kernels are learned from Data.
However it does not mean CNNs do not need any prior: they actually do but they are not at "Data Level" but at the "Learning Machine Level" hence they regard the Architecture related Priors (e.g. number of layers, connections, ...) and Convolutional Kernel related Priors (e.g. size, stride, ...)
