Why people prefer neural network to kernel methods? I am learning Kernel methods. Kernel methods are less a "black box" than neural networks.
Nowadays, it seems neural networks gain more popularity and show more powerful in various applications, such as image, RL. Can anyone give some ideas or literature on the advantage of the networks over kernel? Or are there any deficits of kernel methods?
I guess the easy implementation and intuitive of the network may be an advantage.
 A: I have been asking the same question myself recently. I think intuitively, you can think of kernel methods as obtaining a smooth function that approximates the data, with the smoothness being controlled by a few hyperparameters. The fact that we only have a few parameters to tune suggests that there are constraints in the way the function fits the data. As an example, consider the plot below, where I trained an svm with a Gaussian kernel to classify a cross in a 20x20 dot-matrix. 
Unsurprisingly, it is able to highlight the area with the cross. However, we also see that outside the area of the cross, the surface can be a little "bumpy". This is akin to fitting a cubic spline to a density, where you cannot in general ensure that the predicted values don't fall below zero. I take these as the "constraints" in an SVM. More generally, in higher dimensions, these kinds of constraint limit the way the function approximates the data. For example, it is not possible to have one smoothness parameter for one part of the data and another for another part, unless you define a priori what these parts are. (In the example, I cannot constraint the classifier to have low smoothness around the cross and high smoothness elsewhere.)
I suppose NN don't have these kinds of constraints, as the large number of parameters allow them to take basically any shape they want. Hence, as long as we have a reasonably large amount of data, they outperform kernel methods.
