What do neural networks offer that traditional non-linear statistical models do not offer? I have tried to find an answer to this question but have not found a satisfactory answer. I understand that neural networks(NNs) offer the potential to complex build non-linear models. What I don’t understand is what do NNs offer that traditional statistical non-linear models do not offer. In other words, why would I choose a NN model over a statistical model (e.g. non-linear regression) at least in some cases?
 A: The ability to embed structural and algorithmic priors into the model.
The simplest example of this is convolutional neural networks applied to image data. The structural prior is that nearby regions of the image are more closely related / relevant to each other compared to far-away regions.
Graph convolutional networks extends this "locality" prior to arbitrary graph/network structures. 1D and 3D convolutional networks extends this prior to sound / 1D signal data and 3D scans respectively.
Powerful quadratic programming solvers have been developed. It is possible to literally embed such a QP solver as part of a neural network, inducing an algorithmic prior which says "find solutions which make use of QP". Value Iteration Networks forces a prior which says "make use of this well known RL algorithm to solve this RL problem".
Computer vision scientists can build 3D geometry into a neural network, enforcing the prior "we live in 3D euclidean space, and here is our camera model" into the architecture of the network.
A: It's my understanding that at this point in time, there is no real solid mathematical reason for why NN have seen as much success as they have. Perhaps that's why you can't find any thing convincing at this time, although there are plenty of heuristic arguments. 
The one proof that this brought up a lot (for good reason) is the "Universal Approximation Theorem"; that is, with enough neurons, any smooth function can be approximated arbitrarily well by a large enough neural network. This suggests that given enough parameters in our NN and enough data, we should be able to get arbitrarily close to the true function we are attempting to approximate. 
However, the Universal Approximation Theorem alone doesn't explain the success of NN's, as NN are definitely not the only type of machine learning/statistical model that have this type of property! For a very simple alternative, you could take a linear model and simply expand the covariates to include non-linear terms and interaction effects. This can also approximate any function given enough expansions. 
Now in the case of linear models, although the Universal Approximation Theorem is true, we can do the math right away to see this becomes much too data hungry to ever be of practical use. For example, suppose we have a model with $k$ covariates. A simple linear model with no parameter expansions requires fitting $k$ coefficients. If we want to include just the first order interaction effects, we are now up to $k^2$ coefficients. While this is a richer set of models than simple linear effects, it's still not that complicated. If we want to include third order effects, this requires $k^3$ coefficients. Note that we haven't even addressed adding non-linear parameter expansions yet. If $k$ is at all large, it becomes obvious that this is not going to work out well for approximating functions in which the covariates have complex interactions. 
So to me, the real question is which kind of models can approximate complex relations from a finite set of data well. I think the paragraph above is fairly convincing that linear models with simple parameter expansions are not the way to go. It's my understanding that the argument for NN's is that (a) there's no convincing argument that they won't work and (b) empirically, they seem to be working quite well in a wide body of problems when one has lots of data and complex interaction of features.
