I have been reading about the history of statistical and machine learning algorithms, and am particularly interested in the reasons as to why neural networks have "replaced" kernel-based methods in many applications.
As I understand, kernel-based methods (e.g., SVM, Gaussian Process) were the dominant choice of machine learning algorithms for many years – even when neural networks existed. I tried to do some readings and find out the reasons :
- Why were kernel-based methods originally developed?
- Why do kernel-based methods "work"?
- Why were kernel-based methods able to perform successfully in real world applications?
Based on these readings, I have identified the following theoretical properties of Kernels which might in part answer the above questions:
Cover's Theorem: Cover's Theorem states that patterns that are difficult to separate in lower dimensions might be better separable when projected into higher dimensions. Thus, kernel-based methods try to learn these projection/mapping functions.
Mercer's Theorem: Supposedly, Mercer's Theorem guarantees the existence of this mapping function.
Kernel Trick and the Representer Theorem: I do not fully understand what these are, but it seems that both of these serve to facilitate the computations required in kernel-based methods.
Kernel-Based Methods as Non-Parametric Models: Non-Parametric Models can be considered to potentially have an infinite number of model parameters. Thus, with regards to the bias-variance tradeoff, when compared to parametric regression models with many terms; kernel-based methods may have the ability to better capture higher levels of complexity within the data without suffering as much from the pitfalls of high variance. Kernel-based methods as such do not have model parameters that need to be explicitly estimated and the machine learning process is about deciding optimal hyperparameter values.
On the other hand, the main theoretical result pertaining to neural networks seems to be the Universal Approximation Theorem. Different versions of the Universal Approximation Theorem exist (e.g., more recent statements of this theorem make claims multilayered and finite depth neural networks) – the original statement of this theorem states that a 2-layered neural network with infinite depth can approximate any function to an arbitrary level of precision:
This is said to be closely related to the Kolmogorov-Arnold Representation Theorem, which states that "every multivariate continuous function can be represented as a superposition of the 2-argument addition and continuous functions of one variable":
But at the end of the day, what lead to neural networks becoming the dominant class of machine learning algorithms and outperforming kernel-based methods in many real world applications? One could argue that this was solely based on empirical reasons and circumstantial evidence, e.g., neural networks just tend to outperform kernel-based methods - but are there any mathematical theorems which can attempt to better explain the downfall of kernel-based methods and the fast-paced rise of neural networks?