In the field of Machine Learning, we are often tasked with optimizing a "noisy function", as compared to a "deterministic function" in classical optimization problems.
This means that we have to "reconstruct" the function we want to optimize based on the observed data we have - since data is said to usually come from a semi-random data generating process; as a result, the function we are interested in optimizing is usually said to have a stochastic component.
This means that had we collected the same data at a different time, it is possible that the data would not be identical to the same data collected at a different point in time - since the function (e.g. Loss Function of a Machine Learning Model) we are interested in optimizing is ultimately based on this data having a random component, (I have heard that) it is important the algorithms we use for optimizing these functions has some inherent ability to be "robust"; meaning that the solution the optimization algorithm returns for this function should (within reason) be similar to the solution for the same function based on slightly different data.
An optimization algorithm that is not "robust" potentially has the ability to return significantly different solutions to the same objective function based on slight variations in the data, and this of course is of concern.
My Question: In terms of High Dimensional Loss Functions that are Non-Convex and expensive to differentiate, I have heard that Evolutionary Algorithms (e.g. Genetic Algorithms, Simulated Annealing, Differential Evolution, etc.) can display considerable advantages compared to Gradient Based Optimization Algorithms (e.g. Gradient Descent) - this is because repeatedly taking derivatives of functions with many variables is usually a lot more "computationally expensive" compared to the operations typically performed by Evolutionary Algorithms (e.g. crossover, mutation) .
But in the case of "noisy optimization" - do Evolutionary Algorithms present any advantages compared to Gradient Based Optimization Algorithms when considering properties such as the "robustness" of solutions??
Can someone please comment on this?