Inspired by AnoE's answer to the problem Can a neural network with only 1 hidden layer solve any problem?, I want to rigorously prove that the sine function over the real line cannot be well approximated by a shallow neural network. So we want to show that $$ \sup_{x\in \mathbb{R}} |f(x) - \sin(x)| \geq 1$$ for any, say, neural network $f$ with 1 hidden layer and ReLU activation function.
Can you give me some ideas on how to approach this problem?
A ReLU network is ultimately a piecewise-linear continuous function. Each neuron in the first hidden layer is just a shifted and scaled ReLU. Taking a linear combination of those produces a piecewise linear function, with (at most) as many hingepoints as there are neurons. Applying ReLU to that can create new hingepoints whenever the function crosses 0, but this is at most once per linear segment, so you end up with at most twice as many hingepoints as neurons.
Having established that, the end behavior is linear. If the slope is nonzero, the function goes to infinity, and the supremum of errors is also infinite; if the slope is zero, the supremum of errors is at least 1.
... a shallow neural network
Actually, this isn't an important assumption. Adding layers can increase the number of hinge points multiplicatively, but at the end you're still stuck with a finite number of hinges and hence a bounded good-estimation range.
The classical (Cybenko) universal approximation theorem has a condition about the function being approximated on a compact space.
On the real line, the Heine-Borel theorem says that compacts sets are the closed and bounded sets.
Therefore, the Cybenko universal approximation theorem does not apply to a function over the whole real line. If you approximate $\sin(x)$ over a compact space, such as a closed interval, then the theorem holds.
