I've been reading Chris Bishop's book, Neural Networks for Pattern Recognition recently. It is quite dated relative to the state of the art, but it is the best book I've found on neural nets that actual relates things back to statistics, optimization, etc.
Reading it today, I crossed a passage that I've heard before, specifically: a three layer network (2 hidden layers, or 3 layers of weights) with sigmoidal activation functions can approximate, to arbitrary accuracy, any smooth mapping. He goes on further to say that given an adequate number of hidden units, dimensionality of input to output mapping is not limiting. I believe also he describes a 3 layer network as a universal function approximator.
Also, I recently read a paper (https://arxiv.org/abs/1806.06850) showing empirically that polynomial regression was able to perform as well or better than neural networks of depth similar to the degree of the polynomial, but, they only explored to the power of 3.
With all the interest in deep neural networks, I'm wondering why tall neural networks haven't also been of interest. Obviously the name isn't as sexy, but it seems in general I could have a more shallow network that is simply taller and get to the same result. The only thing that comes to mind for me is the gradients for each layer are much higher dimension, and maybe convergence would be a problem.
Or I'm missing something more deep. I appreciate any thoughts on the topic.