Recently, I was looking at the optimization functions required in training Kernel Based Methods compared to Neural Networks.
1) Kernel Methods:
For instance, I was looking at the optimization in Support Vector Machines:
And Gaussian Process Regression:
2) Neural Networks:
My Question: We often hear the reason that Neural Networks were initially less popular than Kernel Based Methods is because (deep) Neural Networks typically require significantly more computational resources to train compared to Kernel Based Methods.
I have informally heard that Gaussian Process Regression scales better to larger data sets (based on a choice of kernel function, the data can be directly entered into the structural form of the Gaussian Process), and I have also informally heard that training Neural Networks are generally considered to be extremely computationally expensive - but just by looking at the functions associated with each model that require to be optimized, how can we understand the differences in computational costs between Kernel Based Methods and Neural Networks?