Does the universal approximation theorem for neural networks hold for any activation function (sigmoid, ReLU, Softmax, etc...) or is it limited to sigmoid functions?
Update: As shimao points out in the comments, it doesn't hold for absolutely any function. So for which class of activation functions does it hold?
The wikipedia article has a formal statement.
Let $\varphi$ be a nonconstant, bounded, and continuous function.
Multilayer feedforward networks is a published reference that address the issue. Polynomial activation functions do not have the universla approximation property.
The preprint NN with unbounded activation functions covers many activation functions. It looks only at single hidden layer NN. It is heavy on Fourier analysis.
I emphasize that the second reference is a pre-print because I cannot vouch for its accuracy. Leshno et alt 1993 is a reviewed publication.
Kurt Hornik's 1991 paper "Approximation Capabilities of Multilayer Feedforward Networks" proves that "standard multilayer feedforward networks with as few as a single hidden layer and arbitrary bounded and nonconstant activation function are universal approximators with respect to $L^P(\mu)$ performance criteria, for arbitrary finite input environment measures $\mu$, provided only that sufficiently many hidden units are available." In other words, the hypothesis that the activation function is bounded and nonconstant is sufficient to approximate nearly any function given we can use as many hidden units as we like in the neural network. The paper should be available here: http://zmjones.com/static/statistical-learning/hornik-nn-1991.pdf
We have to distinguish between Shallow Neural Networks (one hidden layer) and Deep Neural Networks (more than one hidden layer) since there is a difference.
What I write below can also be found on the Wikipedia page Universal Approximation Theorem.
Shallow Neural Networks: Pinkus showed in 1999 that Shallow Neural Networks with a continuous activation function have the universal approximation property on a compact set $K\subseteq \mathbb{R}$ if and only if the activation function is non-polynomial. The same article mentions that some discontinuous functions can also be used as activation function while preserving the universal approximation property for the networks.
Deep Neural Networks: There are multiple different results. One of them is by Kidger and Lyon and is from 2020. Here they show that Deep Neural Networks have the universal approximation property on a compact set $K\subseteq\mathbb{R}$ when their activation function is:
This shows one of the differences between Deep and Shallow Neural Networks, namely that Deep Neural Networks still have the universal approximation property when their activation function is a (nonaffine) polynomial.
In the article, Kidger and Lyon extend the result in multiple ways. For instance, they show that the result still holds for some activation functions which are continuous but nowhere differentiable.
