2

As title says I would like to understand why there are so many tutorials and notebooks showing how to fit 1 dimensional functions with neural networks instead of polynomial regression. What are the advantages of using MLPs? do they generalize better? 1re they more powerful function approximators?

1
  • 6
    A reason might be that they are tutorials, designed not necessarily for better predictive performance, but to illustrate and teach the basics of neural networks. For which you would use a simple dataset. Commented Nov 14, 2022 at 9:54

1 Answer 1

4

Neural networks are not all that special. Yes, there are these universal approximation theorems saying that, given a decent function, a neural network can approximate it as close as you require. This sounds great until you realize:

  1. The Stone-Weierstrass theorem says about the same for polynomial approximations

  2. Carleson’s theorem says about the same for approximating functions using Fourier series

  3. No universal approximation theorem (and neither Stone-Weierstrass nor Carleson) says how neural networks perform in the presence of noise (the regression error term).

Consequently, neural networks could be considered somewhat overrated (and I say this as someone who likes them and thinks they’re cool).

Cynically, I think people are a bit mesmerized by neural networks. They have seen specific architectures like convolutional neural networks have great success at image recognition, and they want those super-high accuracy scores for their own problems. That makes training in neural networks to be in demand, so tutorial-makers give the people what they want.

Polynomial regression as an alternative to neural nets is a provocative paper that is worth reading.

Cheng, Xi, et al. "Polynomial regression as an alternative to neural nets." arXiv preprint arXiv:1806.06850 (2018).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.