Genetic algorithms are one form of optimization method. Often stochastic gradient descent and its derivatives are the best choice for function optimization, but genetic algorithms are still sometimes used. For example, the antenna of NASA's ST5 spacecraft was created with a genetic algorithm:
When are genetic optimization methods a better choice than more common gradient descent methods?
Genetic algorithms (GA) are a family of heuristics which are empirically good at providing a decent answer in many cases, although they are rarely the best option for a given domain.
You mention derivative-based algorithms, but even in the absence of derivatives there are plenty of derivative-free optimization algorithms that perform way better than GAs. See this and this answer for some ideas.
What many standard optimization algorithms have in common (even derivative-free methods) is the assumption that the underlying space is a smooth manifold (perhaps with a few discrete dimensions), and the function to optimize is somewhat well-behaved.
However, not all functions are defined on a smooth manifold. Sometimes you want to optimize over a graph or other discrete structures (combinatorial optimization) -- here there are dedicated algorithms, but GAs would also work.
The more you go towards functions defined over complex, discrete structures, the more GAs can be useful, especially if you can find a representation in which the genetic operators work at their best (which requires a lot of hand-tuning and domain knowledge).
Of course, the future might lead to forget GAs altogether and develop methods to map discrete spaces to continuous space, and use the optimization machinery we have on the continuous representation.
Genetic methods are well suited for multicriteria optimization when gradient descent is dedicated to monocriteria optimization. Gradient descent allow to find minimum of functions when derivatives exists and there is only one optimum solution (if we except local minimas). A genetics algorithm can be used in multicriteria problems and lead to a continuum of solutions, each one beeing individuals of a population, having evolved from a initial population. The values to optimize are the phenotypes of the individuals and there can be several phenotypes. Generaly, none of the individual has simultaneously the better value of each phenotype, so there is not only one solution. The individuals in the final population, that are all solutions of the optimization, are part of the "Pareto front" and marked as being "Pareto rank one" individuals. This mean that compared to every others individuals having the same performance for each phenotype, they are at least better for one phenotype than the others.
Best in which sense ?
In my experience, GAs are one of the most pragmatic optimizers. While many more precise algorithms require time and effort to formalize real problems in the mathematical world, GAs can handle any cost function with complex rules and constraints (GAs are related by an execution approach afterall and not by specific calculation). This process is straightforward and you can try many approaches for exploratory work.
I appreciate also the possibility to reinject past solutions to the algorithm for future runs which is good for repeated task.
Conceptually, a genetic algorithm can be represented by a hashmap of functions and suits so functionnal languages well like Clojure which is also a language where you can achieve big results very quickly.
Genetic Algorithms can also be nested : the cost function of one GA can be a GA ! These algorithms take advantage of modern hardware and infrastructure which allow them to compute a very large population so that - even with simple mutation/selection operations - you can still achieve good results.
Even for simple problems like finding the minimum of a wave function, GAs are not that bad and can achieve a decent precision in an acceptable time.
So yeah, analytical solutions may have quicker execution time and precision, but the time required to produce them overweights often expected benefits ! So when ? Almost everytime to me, at least for meta-optimization.
Genetic algorithms are best when many processors can be used in parallel. and when the object function has a high modality (many local optima). Also, for multi-objective optimization, there are multi-objective genetic algorithms, MOGA.
However, I think Genetic algorithms are overrated. A lot of the popularity probably comes from the fact that they are inspired by biology. They do tend to find global optima more often than other direct search methods, but at a high cost, which means that another direct search method (e.g. Nelder-Mead, Complex or SQP) can be run several times to find a global optimum, and yield a better performance (i.e., in terms of convergence, and probability of finding global optimum) overall. This is true not only if a single processor is used, but also even when multi processors are available.
I think they stay popular because there are so many problems where very simple models are used, so performance is not an issue.
