4
$\begingroup$

Recently, I was looking at the optimization functions required in training Kernel Based Methods compared to Neural Networks.

1) Kernel Methods:

For instance, I was looking at the optimization in Support Vector Machines:

enter image description here

And Gaussian Process Regression:

enter image description here

2) Neural Networks:

enter image description here

My Question: We often hear the reason that Neural Networks were initially less popular than Kernel Based Methods is because (deep) Neural Networks typically require significantly more computational resources to train compared to Kernel Based Methods.

I have informally heard that Gaussian Process Regression scales better to larger data sets (based on a choice of kernel function, the data can be directly entered into the structural form of the Gaussian Process), and I have also informally heard that training Neural Networks are generally considered to be extremely computationally expensive - but just by looking at the functions associated with each model that require to be optimized, how can we understand the differences in computational costs between Kernel Based Methods and Neural Networks?

Thanks!

References:

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
5
$\begingroup$

Neural networks are not per se expensive to train, but as you can arbitrarily / without end stack layers and layers, you can end up with quite a large network.

A famous NLP network, GPT-3, is quite large:

GPT-3 comes in eight sizes, ranging from 125M to 175B parameters

Further, for achieving good results with NN you usually use huge amounts of training data, such as in this case

Notice GPT-2 1.5B is trained with 40GB of Internet text, which is roughly 10 Billion tokens (conversely assuming the average token size is 4 characters)

That even is a problem with storage, because

The 175 Billion parameters needs 175 × 4 = 700 G B memory to store in FP32 (each parameter needs 4 Bytes).

The point I want to make is that successful NN are often deep and trained with lots of data, that does not mean that the class of algorithms per se is expensive to train.

All quotes are from https://lambdalabs.com/blog/demystifying-gpt-3/

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ (+1) Another concept of expensive is the dollar cost of training the model. "Lambda Labs calculated the computing power required to train GPT-3 based on projections from GPT-2. According to the estimate, training the 175-billion-parameter neural network requires 3.114E23 FLOPS (floating-point operation), which would theoretically take 355 years on a V100 GPU server with 28 TFLOPS capacity and would cost \$4.6 million at \$1.5 per hour." $\endgroup$
    – Sycorax ♦
    Commented Jan 28, 2022 at 19:54
  • $\begingroup$ @ Nikolas: Thank you so much for your answer! And are we able to compare these costs to training "Kernel Based Methods" (e.g. SVM) on a similar data set/problem? Thank you so much! $\endgroup$
    – stats_noob
    Commented Jan 28, 2022 at 20:29
  • $\begingroup$ @ Sycorax: WOW - deep learning is making a DEEP hole in my bank account! LOL $\endgroup$
    – stats_noob
    Commented Jan 28, 2022 at 20:29
  • $\begingroup$ Another important aspect that differs between say SVM and NN is that NN are never finished with training. Training is an incremental process in which all weights are adapted by backpropagation. You can always train one more step. That is not the case for SVM (and most other algorithms). $\endgroup$
    – Nikolas Rieble
    Commented Jan 29, 2022 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.