I know this is a loaded question given the infinite number of circumstances surrounding what kind of machine learning algorithm to implement. I was just wondering if there is a general framework that can hint at a situation in which "regular" machine learning algorithm will certainly outperform a neural net.
In the case of a normally distributed response with normally distributed features, $E(y|x)$ is provably the best minimizer of mean squared predictive error.
See this question.
Or Bickel and Doksum text "Mathematical Statistics."