Compare convergence of optimization methods I need to quantify how 2 optimization methods differ in convergence. 
When training a neural network I get the following plots, which show an error function after each gradient update. I think the green one is preferable, because its variance reduces as the learning proceeds. I would like to make more meaningful plot to show that. I was thinking about sort of moving skewness, could you please tell me if it is a good idea or I need something else than skewness? Thank you very much! 

 A: Generally, when we build optimization routines we need some stopping criteria. For instance, we may look the last change of the target function: $\Delta y_i=y_i-y_{i-1}$, where $y_i$ is the value of the target function at iteration $i$. Whenever $|\Delta y_i|<c$ we stop, where $c$ is a threshold. This would guarantee that you reach the desired precision. 
Another criteria is a relative tolerance: $\partial y_i= \frac{\Delta y_i}{y_i}$ and $|\partial y_i|<c_r$. Often we combine these two, and stop when either one of them is reached.
Finally, we usually add a max number of iterations constraint, and stop the optimization when it's reaching additionally throwing a warning that stopping criteria were not reached at max number of iterations.
It seems that you want to go backwards in this waterfall. So, what you can do is to stop at some number of iterations, and collect $\Delta y_i$ and $\partial y_i$ to compare the methods.
Instead of using $\Delta y_i$ you could compute some sort of summary metrics such as moving absolute average $\sum_{j=0}^{n-1} |\Delta y_{i-j}|$ or moving variance. I wouldn't bother though, because in the end it's not going to change your conclusions, you'll see.
