Does the universal approximation theorem for neural networks hold for any activation function?

Does the universal approximation theorem for neural networks hold for any activation function (sigmoid, ReLU, Softmax, etc...) or is it limited to sigmoid functions?

Update: As shimao points out in the comments, it doesn't hold for absolutely any function. So for which class of activation functions does it hold?

• I believe it holds for all those you listed, but it doesn't hold for any arbitrary activation function (consider f(x) = 0) Jan 30 '18 at 5:07
• Read the Cybenko (1989) paper. The function has to be compact i.e needs to be defined on compact subsets of R^n Sep 3 '19 at 14:51
• If there are finitely many discontinuities, it can be handled as well by adding more hidden layers. It works for SBAF as well. Sep 3 '19 at 14:58
• This makes little sense, because every function defined on $\mathbb{R}^n$ is defined on compact subsets of it!
– whuber
Sep 3 '19 at 15:17

The wikipedia article has a formal statement.

Let $$\varphi$$ be a nonconstant, bounded, and continuous function.

• That covers sigmoid and softmax but not ReLU. According to this paper the property also holds for some some unbounded functions like ReLU as well as others. Jan 30 '18 at 5:33

Multilayer feedforward networks is a published reference that address the issue. Polynomial activation functions do not have the universla approximation property.

The preprint NN with unbounded activation functions covers many activation functions. It looks only at single hidden layer NN. It is heavy on Fourier analysis.

I emphasize that the second reference is a pre-print because I cannot vouch for its accuracy. Leshno et alt 1993 is a reviewed publication.

Kurt Hornik's 1991 paper "Approximation Capabilities of Multilayer Feedforward Networks" proves that "standard multilayer feedforward networks with as few as a single hidden layer and arbitrary bounded and nonconstant activation function are universal approximators with respect to $$L^P(\mu)$$ performance criteria, for arbitrary finite input environment measures $$\mu$$, provided only that sufficiently many hidden units are available." In other words, the hypothesis that the activation function is bounded and nonconstant is sufficient to approximate nearly any function given we can use as many hidden units as we like in the neural network. The paper should be available here: http://zmjones.com/static/statistical-learning/hornik-nn-1991.pdf