When it comes to the Loss Functions of "Real World" Statistical Models (e.g. Neural Networks) - based on the large number of parameters in these loss functions, their complex behaviors and the fact that they are based on non-deterministic variables (i.e. random variables) : I have heard that it is reasonable to believe that many of these loss functions are non-convex and noisy.
I have heard that the desirable properties (e.g. convergence) of gradient based optimization algorithms do not necessarily extend themselves fully to non-convex and non-deterministic (i.e. "noisy") functions. Compounded by the fact that even Quasi-Newton optimization techniques (e.g. such as gradient descent, that do not rely on the evaluating the second derivatives of the loss function) can be quite computationally expensive for such types of loss functions - does this explain the rise in popularity of evolutionary algorithms (e.g. genetic algorithm) and metaheuristics for optimizing loss functions?
My Question: In problems where we actually have a loss function (as opposed to discrete combinatorial optimization problems where we don't have loss functions) - compared to gradient based optimization techniques, do evolutionary algorithms present advantages for the following reasons:
Optimizing complex and high dimensional loss functions with many model parameters (i.e. the weights in a neural network) make gradient based optimization techniques (e.g. gradient descent) computationally expensive based on the fact that they have to repeatedly evaluate derivatives of the loss function - whereas Evolutionary Algorithms do not require evaluating derivatives of the loss functions, making them more suitable in such circumstances.
In general, Evolutionary Algorithms offer no theoretical guarantees on convergence whatsoever - they only guarantee improvements in successive iterations. Seeing as we are still uncertain if the desirable properties of gradient based algorithms (e.g. convergence) extend themselves to non-convex and noisy functions - did this fact serve as one of the reasons as to why Evolutionary Algorithms gained popularity in optimizing machine learning models? Why exactly do Evolutionary Algorithms present advantages in optimizing noisy functions?
Can someone please comment on this?