How to compare two different algorithms for deep RL? Assume we have two algorithms for a deep RL problem. We used them to train an agent for this environment and stored the rewards at each episode for these two different algorithms. My question is that how can we say which algorithm is better?
Plot for all 50k episodes

Plot for 50k episodes where we have only plotted once every 20 episodes

Plot for 50k episodes where we have only plotted once every 200 episodes

Another question, which plot is the one people use when they want to plot reward\return with respect to the episode numbers?
 A: 
My question is that how can we say which algorithm is better?

It doesn't look to me like you can say which algorithm is better from these experiments at all; the only reasonable conclusion I'd be able to draw from these plots is that they perform pretty much identically, neither is clearly better or worse than the other.

Some things you can try to get more informative plots:


*

*Multiple runs of 50K episodes. If I understand your post correctly, you ran one sequence of 50K episodes for each of the algorithms, and plot those results. RL methods, in particularl Deep RL ones, are known to be susceptible to having wildly varying performance levels just based on initial random seeds. Therefore, it would be useful have multiple "runs", starting at episode = 0 and ending at e.g. episode = 50K for each algorithm (meaning, using the same algorithm to re-start learning completely from scratch multiple times). For every point on your $x$-axis (every episode number), you can then plot an average performance level across all runs, rather than just a single measure of performance. This will likely reduce the "spikiness" in your plots a bit.

*Plot running averages. Another cause of the "spikiness" in your plots is that you are likely using a policy that involves some element of exploration, like an $\epsilon$-greedy strategy. Even when an algorithm has learned what the optimal policy would be, it will sometimes intentionally do something different instead for the sake of exploration, and this can lead to occasional poor performances even after an algorithm has already demonstrated to be capable of performing better. You can smoothen this out by plotting running averages, rather than single point estimates of performance. What this means is, rather than plotting the score obtained in episode $x$ for every episode $x$, you can plot the average score obtained during the window of episodes $[x-50, x]$ for every episode $x$. Of course that's just an example with a window size of $50$, you can also use different window sizes.

*Plot standard deviation or 95% confidence intervals (or 99% confidence intervals or whatever you choose). These can be drawn as transparant, shaded areas around measured means of performance. This will only really work in combination with one or both of the points above; standard deviations or confidence intervals for single point estimates don't really make sense. If the shaded areas for two different algorithms overlap for a large part, that's indicative of the algorithms having similar performance levels, even if the means may be slightly different.

If I look at your current plots though, I highly doubt any of the ideas above are going to enable you to conclusively say one algorithm outperforms the other; they really do seem to perform pretty much the same. Of course, one may be better than the other in some different dimension though (faster runtime, less memory, easier to understand / implement, fewer hyperparameters, easier hyperparameters to tune / understand, etc.). The paper I linked above (Deep Reinforcement Learning that Matters) is probably a good one to read through for more information other than what I mentioned above too.
