From my understanding, neural networks are universal approximators, which was proven by Cybenko in 1989 for sigmoid activation functions with output units being linear. Castro showed, that this property also holds true even if the output is not linear, but is a squashing function (non-decreasing with values in $[0,1]$). But did someone manage to prove that this property is also true for modern NNs with ReLU activations? I'm quite new to this topic and I haven't been able to find such work, I'd be grateful for your help.
1 Answer
$\begingroup$
$\endgroup$
You might find some results in "OPTIMAL APPROXIMATION WITH SPARSELY CONNECTED DEEP NEURAL NETWORKS" by Bölcskei et al. (download)
They have a general approximation theoretic approach to modern networks and show results in approximation quality, also using ReLU etc...
From their abstract: "Our central result elucidates a remarkable universality property of neural networks and shows that they achieve the optimum approximation properties of all affine systems combined"
