I read most of the paper and when you read the second paragraph it gives you the idea that they missed the central problem. I have run into this problem myself in another field. A parametric model will underperform a non-parametric model when the parametric model is misspecified. If it is badly misspecified then the non-parametric version will be vastly superior. From what I am reading in the article, you are looking at badly misspecified parametric models.
This is pretty serious in itself because it implies the theories in the field are either very off or when they are being parameterized some important element of statistical theory is being applied incorrectly.
There is another danger I see in the paper; it is the presumption that neural networks map to non-parametric solutions when some neural networks actually map to parametric solutions. They appear non-parametric because the distribution function isn't specified, but it can be implicit in the form. I can think of cost functions that would exclude certain distributions from being present, or if they were present it would prevent learning.
My guess is that there is a serious mismatch somewhere between the math in use and the math implied by nature but no one has fully investigated it. If I were in this field, it would cause me to go back to the base models and ask why they were built the way they were and if something is present in the environment that is causing parametric model failure. This is challenging because it is likely that there is either an implicit assumption or a missing constraint that parametric models should account for in a different manner.
This neural network model is a small incremental gain to the field. Tracking down the source of the discrepancy would move our understanding of deep learning. It may be something that is very important.
It reminds me of a story from an engineering professor. There was a recipe a young engineer was using for meatloaf that required cutting all four sides of the meatloaf prior to serving. It was trimmed down. Her mother had always done that. Eventually, she asked her mother why she did it. Her mother said that the pan she baked the meatloaf in was larger than the pan she served it in, so she wrote down the instruction of how to trim it so she would remember the dimensions of the smaller serving pan.
You are looking for something that everyone is taking for granted. If you find it, then it will probably be obvious to everyone what should have happened. Most things like this are so obvious no one can see it.
Pull the very first set of articles on these methods. Walk through the mental process used to construct them. Figure out what they imply even if it isn't obvious. Maybe you need to quit cutting the meatloaf to fit a pan you are not using, or maybe you are trying to serve an oversized meatloaf in an undersized pan but nobody has figured out to measure and cut yet.
