3
$\begingroup$

I am learning Kernel methods. Kernel methods are less a "black box" than neural networks.

Nowadays, it seems neural networks gain more popularity and show more powerful in various applications, such as image, RL. Can anyone give some ideas or literature on the advantage of the networks over kernel? Or are there any deficits of kernel methods?

I guess the easy implementation and intuitive of the network may be an advantage.

Improve this question
$\endgroup$
3
  • $\begingroup$ I did not remember where I read the following claims. The kernel methods may work better in the middle-size dataset. But neural networks show a huge advantage on large datasets empirically. $\endgroup$
    – Qinsheng Zhang
    Jun 17, 2020 at 2:53
  • 1
    $\begingroup$ This thread is an interesting read, in hindsight. stats.stackexchange.com/questions/30042/… Also: stats.stackexchange.com/questions/366581/… $\endgroup$
    – Sycorax ♦
    Jun 17, 2020 at 3:40
  • 1
    $\begingroup$ "Nowadays, it seems neural networks gain more popularity" popularity is not necessarily based on good reason - a lot of it is fashion/hype. Neural nets as a field have been through multiply hype-bust cycles during my career, where the tasks for which they are suitable become interesting for a while, and then people over-apply them because of the hype and find them dissapointing. Best to have lots of varied tools in your toolbox and know how to choose the right tool and apply them well. $\endgroup$
    – Dikran Marsupial
    Apr 18 at 13:35

2 Answers 2

Reset to default
2
$\begingroup$

I have been asking the same question myself recently. I think intuitively, you can think of kernel methods as obtaining a smooth function that approximates the data, with the smoothness being controlled by a few hyperparameters. The fact that we only have a few parameters to tune suggests that there are constraints in the way the function fits the data. As an example, consider the plot below, where I trained an svm with a Gaussian kernel to classify a cross in a 20x20 dot-matrix. 1

Unsurprisingly, it is able to highlight the area with the cross. However, we also see that outside the area of the cross, the surface can be a little "bumpy". This is akin to fitting a cubic spline to a density, where you cannot in general ensure that the predicted values don't fall below zero. I take these as the "constraints" in an SVM. More generally, in higher dimensions, these kinds of constraint limit the way the function approximates the data. For example, it is not possible to have one smoothness parameter for one part of the data and another for another part, unless you define a priori what these parts are. (In the example, I cannot constraint the classifier to have low smoothness around the cross and high smoothness elsewhere.)

I suppose NN don't have these kinds of constraints, as the large number of parameters allow them to take basically any shape they want. Hence, as long as we have a reasonably large amount of data, they outperform kernel methods.

Improve this answer
$\endgroup$
2
$\begingroup$

To my mind there are a few reasons for this:

  • We don't know how to train kernel methods efficiently on datasets are large as we can for neural networks
  • Similarly, neural networks allow more parallel computation
  • Mathematically, many kernel methods (such as kernel ridge regression) satisfy a representer theorem, which says that the learned function lives in the span of a finite set of evaluations of the kernel. I would guess that this is quite restrictive, especially in high dimensions (e.g., for inner product kernels), and that neural networks don't have this drawback.

(It's worth saying that what I've called a drawback above for kernel machines is actually a great strength when it comes to theoretical analysis. It might just not help practically)

Improve this answer
$\endgroup$
19
  • $\begingroup$ "I would guess that this is quite restrictive" quite the opposite, especially if your kernel is one with an infinite dimensional feature space and gives rise to universal approximation. Note there is a kernel that is very much like a sigmoidal neuron and very large neural networks (at least shallow ones) have been shown to be equivalent to Gaussian processes, which are (Bayesian) kernel methods. The representer theorem means you have a much better chance of finding the optimal solution. For neural networks we often rely on finding a good local minima to avoid overfitting - bit of a hack! $\endgroup$
    – Dikran Marsupial
    Apr 18 at 13:32
  • $\begingroup$ You are correct about universal approximation. In fact, it's not too hard to show that if the kernel function is strictly positive definite then the RKHS is dense in the space of continuous functions with respect to (IIRC) the uniform topology. However, this guarantees that approximation is possible in principle, not in practice. Moreover, for infinite dimensional RKHSs, for any finite training samples there is an infinite dimensional space that's orthogonal to your learned predictor... $\endgroup$
    – user27182
    Apr 18 at 13:52
  • $\begingroup$ ...The corresponding component of the target function can be shown to give a lower bound on the generalisation error in kernel regression. AFAIK the same issue does not seem to exist for NNs. By the way, the NNGP correspondence holds for deep networks too arxiv.org/abs/1711.00165 $\endgroup$
    – user27182
    Apr 18 at 13:53
  • $\begingroup$ I should say: I don't have any expertise (or much knowledge at all!) in approximation theory of either NNs or kernel machines. It could be that kernels have just as good approximation properties (e.g., in terms of rate, dependence on intrinsic data dimensionality, etc.), but this isn't so relevant to my comment about generalisation and the finite dimensional span. $\endgroup$
    – user27182
    Apr 18 at 14:00
  • $\begingroup$ "However, this guarantees that approximation is possible in principle, not in practice. " I don't see how that could be any different for NNs, especially as the NN has a non-convex loss function. Yes, obviously the Gram matrix describes a subspace of the infinite dimensional feature space, which is why they can be evaluated with finite maths, but the orthogonal space is irrelevant to minimising the cost function, so it is not clear why that is a problem. Thanks for the link to the report - doesn't it suggest though that GPs are more capable than DNNs (ignoring computational expense)? $\endgroup$
    – Dikran Marsupial
    Apr 18 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.