Does the universal approximation theorem apply to ReLu? Hornik (1991) states that when the activation function is continuous, bounded and non-constant, a single hidden layer suffices to approximate any continuous function. From my understanding this doesn't apply to ReLu, since it is not bounded. Does this mean the universal approximation theorem doesn't apply to ReLu?
 A: I believe the second-to-last paragraph of page 253 gives a method for making corollaries involving ReLU networks: take three ReLU neurons to generate a "spike" activation function:
$s(x) := \DeclareMathOperator{relu}{ReLU} \relu(x-1) - 2 \relu(x) + \relu(x+1)$

(W|A link)
This function is bounded, continuous, and non-constant and so satisfies theorems 1 and 2 of the paper (but discontinuous derivatives, so not theorems 3 and 4); and the space of relu-network functions contains the space of "spike"-network functions by grouping neurons into threes as above.
A: The function being approximated is what must be bounded, not the functions in the nodes (activation functions), so ReLU fits in the universal approximation theorem framework.
(The term you might be more likely to see in the discussion of the function being approximated is “compact”. The Heine-Borel theorem in real analysis says that, in Euclidean space ($\mathbb{R}^n$ with the usual notion of distance), a “compact” set is the same as one that is closed and bounded.)
