Does the universal approximation theorem for neural networks hold for any activation function (sigmoid, ReLU, Softmax, etc...) or is it limited to sigmoid functions?

Update: As shimao points out in the comments, it doesn't hold for absolutely any function. So for which class of activation functions does it hold?

  • 1
    $\begingroup$ I believe it holds for all those you listed, but it doesn't hold for any arbitrary activation function (consider f(x) = 0) $\endgroup$
    – shimao
    Jan 30, 2018 at 5:07
  • $\begingroup$ Read the Cybenko (1989) paper. The function has to be compact i.e needs to be defined on compact subsets of R^n $\endgroup$ Sep 3, 2019 at 14:51
  • $\begingroup$ If there are finitely many discontinuities, it can be handled as well by adding more hidden layers. It works for SBAF as well. $\endgroup$ Sep 3, 2019 at 14:58
  • $\begingroup$ This makes little sense, because every function defined on $\mathbb{R}^n$ is defined on compact subsets of it! $\endgroup$
    – whuber
    Sep 3, 2019 at 15:17

4 Answers 4


The wikipedia article has a formal statement.

Let $\varphi$ be a nonconstant, bounded, and continuous function.

  • 10
    $\begingroup$ That covers sigmoid and softmax but not ReLU. According to this paper the property also holds for some some unbounded functions like ReLU as well as others. $\endgroup$
    – jodag
    Jan 30, 2018 at 5:33

Multilayer feedforward networks is a published reference that address the issue. Polynomial activation functions do not have the universla approximation property.

The preprint NN with unbounded activation functions covers many activation functions. It looks only at single hidden layer NN. It is heavy on Fourier analysis.

I emphasize that the second reference is a pre-print because I cannot vouch for its accuracy. Leshno et alt 1993 is a reviewed publication.


Kurt Hornik's 1991 paper "Approximation Capabilities of Multilayer Feedforward Networks" proves that "standard multilayer feedforward networks with as few as a single hidden layer and arbitrary bounded and nonconstant activation function are universal approximators with respect to $L^P(\mu)$ performance criteria, for arbitrary finite input environment measures $\mu$, provided only that sufficiently many hidden units are available." In other words, the hypothesis that the activation function is bounded and nonconstant is sufficient to approximate nearly any function given we can use as many hidden units as we like in the neural network. The paper should be available here: http://zmjones.com/static/statistical-learning/hornik-nn-1991.pdf


We have to distinguish between Shallow Neural Networks (one hidden layer) and Deep Neural Networks (more than one hidden layer) since there is a difference.

What I write below can also be found on the Wikipedia page Universal Approximation Theorem.

Shallow Neural Networks: Pinkus showed in 1999 that Shallow Neural Networks with a continuous activation function have the universal approximation property on a compact set $K\subseteq \mathbb{R}$ if and only if the activation function is non-polynomial. The same article mentions that some discontinuous functions can also be used as activation function while preserving the universal approximation property for the networks.

Deep Neural Networks: There are multiple different results. One of them is by Kidger and Lyon and is from 2020. Here they show that Deep Neural Networks have the universal approximation property on a compact set $K\subseteq\mathbb{R}$ when their activation function is:

  • Continuous
  • Nonaffine (i.e. not a multivariate polynomial of degree less than 2)
  • Differentiable with a continuous derivative which is different from 0 at at least one point (the exact technical formulation is slightly more strict).

This shows one of the differences between Deep and Shallow Neural Networks, namely that Deep Neural Networks still have the universal approximation property when their activation function is a (nonaffine) polynomial.

In the article, Kidger and Lyon extend the result in multiple ways. For instance, they show that the result still holds for some activation functions which are continuous but nowhere differentiable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.