analysis of optimal neural nets? Neural nets are trained with learning algorithms that are very very unlikely to find the global optimum in parameter space (i.e. the global optimum neural net out of the set of possible neural nets given the neural architecture).
Nevertheless, I am wondering if there is an analysis of what these optimal neural nets would look like, conditional on the architecture and the problem


*

*how good would the optimal neural net given architecture $A$ and problem $P$ be?

*what would its properties be?
Note, I'm not asking for specific answers to those questions, but whether there is a literature that is concerned with this.
 A: The results of this paper seem highly relevant.
Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, Xiyu Zhai. "Gradient Descent Finds Global Minima of Deep Neural Networks."

Gradient descent finds a global minimum in training deep neural networks despite the objective function being non-convex. The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). Our analysis relies on the particular structure of the Gram matrix induced by the neural network architecture. This structure allows us to show the Gram matrix is stable throughout the training process and this stability implies the global optimality of the gradient descent algorithm. Our bounds also shed light on the advantage of using ResNet over the fully connected feedforward architecture; our bound requires the number of neurons per layer scaling exponentially with depth for feedforward networks whereas for ResNet the bound only requires the number of neurons per layer scaling polynomially with depth. We further extend our analysis to deep residual convolutional neural networks and obtain a similar convergence result.

