When we test a new optimization algorithm for a particular problem at hand, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 score .etc, and do the same for an old optimization algorithm, or do we need to compute the average performance,i.e.,the average value of accuracy or f1 scores for these runs, to show that it is better than the old optimization algorithm? Because when I read the papers on a new optimization algorithm, I don't know how they calculate the performance and draw the train-loss vs iters curves, because it has random effects, and for different runs we may get different performance and different curves. So do we compare best performance or average performance?
I can only speak for SGD-variants typically used for neural networks.
Typically it's the average over a small number (1-10) of random seeds. Usually if one optimizer is actually better than another, then it will be extremely apparent after just a single run, and the inter-run variance is typically very small, so a larger number of random runs is not useful.