The universal approximation theorem basically states that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $\mathbb{R}^n$.

Let's define a monotone function of vectors to mean any real-valued $f$ such that for any two vectors $\mathbf{x}$ and $\mathbf y$, $$x_i \le y_i \forall i \implies f(\mathbf{x}) \le f(\mathbf{y}).$$ The vector-valued function case is analogous.

Is it true that a feed-forward network with a single hidden layer containing a finite number of neurons, and whose link weights are nonnegative (but whose biases remain unconstrained) can approximate continuous monotonic functions on compact subsets of $\mathbb{R}^n$?

  • $\begingroup$ I don't know the answer but Kurt Hornik ( part of R-core ) is a generous, knowledgable person who will either answer you directly or atleast point you in the right direction. He wrote the famous paper with Hal White whose title escapes me. $\endgroup$
    – mlofton
    Commented Nov 10, 2018 at 6:30

1 Answer 1


I am not sure if a single hidden layer is sufficient, but it can be shown that if your input is in $\mathbb{R}^k$, you will need at most $k$ hidden layers. See Theorem 3.1 in https://ieeexplore.ieee.org/document/5443743

Theorem 3.1: For any continuous monotone nondecreasing function $f : K \rightarrow \mathbb{R}$, where $K$ is a compact subset of $\mathbb{R}^k$, there exists a feedforward neural network with at most $k$ hidden layers, positive weights, and output $O$ such that $|f(\mathbf{x}) - O_{\mathbf{x}}| < \varepsilon$, for any $\mathbf{x} \in K$ and $\varepsilon > 0$.

  • 4
    $\begingroup$ I'll add a bounty of 50 points to thank you for the brilliant find and to welcome you to stats.stackexchange! $\endgroup$
    – Neil G
    Commented Jul 26, 2019 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.