There seems to be two types of research papers in optimization:
papers that characterize the solution of some optimization problem extremely precisely (existence, uniqueness, mixed/continuous/discrete, closed/compact/open, convex/non-convex/saddle...), and then design an algorithm that solves for the exact minimizer of those optimization problems with guaranteed convergence rate, etc.
papers that do not care too much about the structure of the optimization problem, just use some popular method (GD, SGD, SGD + momentum, etc.) to update some parameters.
It seems that almost all the deep learning, supervised learning, neural network, autonomous driving applications fall into the second category. No effort is made what so ever to characterize these problems, and when challenged, just say "the problem is highly non-convex and optimization is hard". For example, the ADAM paper is well known to be wrong. And multiple papers have said that SGD + Nesterov + Momentum + ... doesn't really work.
But in practice works pretty well!
I am arguing with someone whether if it is justified to say that all of the success of deep learning comes from the fact we do not care about solving a problem precisely, or even care about whether if our solution is good. The point is, the moment you toss out rigor, and rigor doesn't even seem to matter, your problem becomes pretty easy to solve. It is easy to achieve success on easy problems.
Is this assertion justified?