Does it make sense to use neural network in place of polynomial regression to fit 1 dimensional functions when only 1 feature is available? As title says I would like to understand why there are so many tutorials and notebooks showing how to fit 1 dimensional functions  with neural networks instead of polynomial regression. What are the advantages of using MLPs? do they generalize better? 1re they more powerful function approximators?
 A: Neural networks are not all that special. Yes, there are these universal approximation theorems saying that, given a decent function, a neural network can approximate it as close as you require. This sounds great until you realize:

*

*The Stone-Weierstrass theorem says about the same for polynomial approximations


*Carleson’s theorem says about the same for approximating functions using Fourier series


*No universal approximation theorem (and neither Stone-Weierstrass nor Carleson) says how neural networks perform in the presence of noise (the regression error term).
Consequently, neural networks could be considered somewhat overrated (and I say this as someone who likes them and thinks they’re cool).
Cynically, I think people are a bit mesmerized by neural networks. They have seen specific architectures like convolutional neural networks have great success at image recognition, and they want those super-high accuracy scores for their own problems. That makes training in neural networks to be in demand, so tutorial-makers give the people what they want.
Polynomial regression as an alternative to neural nets is a provocative paper that is worth reading.
Cheng, Xi, et al. "Polynomial regression as an alternative to neural nets." arXiv preprint arXiv:1806.06850 (2018).
