There are excellent answers, mostly weighing in with the usefulness of DL and ANNs. But I would like to object the OP in a more fundamental way, since the question already takes for granted the mathematical inconsistency of neural networks.

First of all, there is a mathematical theory behind (most models of) Neural Networks. You could likewise argue that linear regression does not generalize, unless the underlying model is... well, linear. In neural algorithms, a model is assumed (even if not explicitly) and the fitting error is computed. The fact that algorithms are modified with various heuristics does not void the original mathematical support. BTW, local optimization is also a mathematically consistent, let alone useful, theory.

Along this line, if Neural Networks just constitute one class of methods within the whole toolbox of scientists, which is the line that separates Neural Networks from the rest of techniques? In fact, SVMs were once considered a class of NNs and they still appear in the same books. On the other hand, NNs could be regarded as a (nonlinear) regression technique, maybe with some simplification. I agree with the OP that we must search better, well founded, efficient algorithms, regardless you label them as NNs or not.