Hornik (1991) states that when the activation function is continuous, bounded and non-constant, a single hidden layer suffices to approximate any continuous function. From my understanding this doesn't apply to ReLu, since it is not bounded. Does this mean the universal approximation theorem doesn't apply to ReLu?
1
-
3$\begingroup$ I may be not correct but ReLu is bounded below and unbounded above. Hornik at least mentions at page 253 in the bottom, that their theorem does not account for all unbounded activation functions. The behavior of an unbounded tail function is also highlighted. So I would "guess" ReLu falls into tha category, where this theorem can not account for all unbounded activation functions $\endgroup$– Patrick BormannCommented May 24, 2021 at 13:22
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I believe the second-to-last paragraph of page 253 gives a method for making corollaries involving ReLU networks: take three ReLU neurons to generate a "spike" activation function:
$s(x) := \DeclareMathOperator{relu}{ReLU} \relu(x-1) - 2 \relu(x) + \relu(x+1)$
(W|A link)
This function is bounded, continuous, and non-constant and so satisfies theorems 1 and 2 of the paper (but discontinuous derivatives, so not theorems 3 and 4); and the space of relu-network functions contains the space of "spike"-network functions by grouping neurons into threes as above.
answered May 24, 2021 at 15:04
$\begingroup$
$\endgroup$
3
The function being approximated is what must be bounded, not the functions in the nodes (activation functions), so ReLU fits in the universal approximation theorem framework.
(The term you might be more likely to see in the discussion of the function being approximated is “compact”. The Heine-Borel theorem in real analysis says that, in Euclidean space ($\mathbb{R}^n$ with the usual notion of distance), a “compact” set is the same as one that is closed and bounded.)
-
4$\begingroup$ Are you sure about that? I think $\psi$ denotes the activation function, and the conditions refer to $\psi$ if I read it correctly... $\endgroup$– krvgerCommented May 24, 2021 at 13:42
-
$\begingroup$ I didn't read the paper. It is possible that the author is discussing stronger conditions, but what I wrote applies to the standard universal approximation theorem. $\endgroup$– DaveCommented May 24, 2021 at 14:02
-
1$\begingroup$ Hornik is interested in fairly strong approximation results over all of $\mathbb{R}^d$, not just over a single compact set. In particular, he notes that polynomials are not universal approximants in the senses he is using, because they can only approximate a few types of tail behaviour as the inputs go to infinity. As @BenReiniger says, ReLU is ok because there are combinations of ReLUs that do satisfy his conditions even though a single ReLU doesn't $\endgroup$ Commented May 21 at 22:40