In DeepMind's paper on Deep Q-Learning for Atari video games (here), there are a lot of hyper parameters that need to be tuned. But, I want to specifically know whether there are some intuitions behind the "final exploration step" (the number of steps over which the value of epsilon is decreased from 1 to 0.01) and "replay memory size"?
1- For the "final exploration step", should we set this number such that during these number of exploration steps, we can be sure that all possible rewards in the environment are observed by the agent? To be more specific, if we assume a gridworld of size 10, where the agent is located at top left corner, the best terminal state is located at bottom right corner, and the agent can move to a neighbor cell at each step, how should we set the "final exploration step"?
2- If Q1 is hard to answer, another related question for the "final exploration step" is that, in the gridworld described above, let's say we have set the "final exploration step" to be 10. In this case, it is impossible for the agent to reach the best terminal state and experience the best reward in the exploration phase. Okay, are we still good meaning that although the exploration phase has been done (and epsilon is now 0.01) and the agent haven't experienced the best terminal state during this phase, is there still a chance that the agent learns about the best terminal state?
3- For the "replay memory size", is there a trade-off here? In the DQN, it is said that we first fill the replay memory with random actions for the agent. Should the "replay memory size" be such that all possible rewards in the environment are observed by the agent and are entered the memory?