As title says I would like to understand why there are so many tutorials and notebooks showing how to fit 1 dimensional functions with neural networks instead of polynomial regression. What are the advantages of using MLPs? do they generalize better? 1re they more powerful function approximators?
-
6A reason might be that they are tutorials, designed not necessarily for better predictive performance, but to illustrate and teach the basics of neural networks. For which you would use a simple dataset.– Stephan KolassaCommented Nov 14, 2022 at 9:54
1 Answer
Neural networks are not all that special. Yes, there are these universal approximation theorems saying that, given a decent function, a neural network can approximate it as close as you require. This sounds great until you realize:
The Stone-Weierstrass theorem says about the same for polynomial approximations
Carleson’s theorem says about the same for approximating functions using Fourier series
No universal approximation theorem (and neither Stone-Weierstrass nor Carleson) says how neural networks perform in the presence of noise (the regression error term).
Consequently, neural networks could be considered somewhat overrated (and I say this as someone who likes them and thinks they’re cool).
Cynically, I think people are a bit mesmerized by neural networks. They have seen specific architectures like convolutional neural networks have great success at image recognition, and they want those super-high accuracy scores for their own problems. That makes training in neural networks to be in demand, so tutorial-makers give the people what they want.
Polynomial regression as an alternative to neural nets is a provocative paper that is worth reading.
Cheng, Xi, et al. "Polynomial regression as an alternative to neural nets." arXiv preprint arXiv:1806.06850 (2018).