14
$\begingroup$

I am trying to understand the famous paper "Playing Atari with Deep Reinforcement Learning" (pdf). I am unclear about the difference between an epoch and episode. In algorithm $1$, the outer loop is over episodes, while in figure $2$ the x-axis is labeled epoch. In the context of reinforcement learning, I'm not clear what an epoch means. Is an epoch an outer loop around the episode loop?

enter image description here

enter image description here

$\endgroup$
  • 1
    $\begingroup$ So...how many episodes make an epoch? $\endgroup$ – Lewen Oct 24 '17 at 16:33
10
$\begingroup$
  • one episode = one a sequence of states, actions and rewards, which ends with terminal state. For example, playing an entire game can be considered as one episode, the terminal state being reached when one player loses/wins/draws. Sometime, one may prefer to define one episode as several games (example: "each episode is a few dozen games, because the games go up to score of 21 for either player").
  • one epoch = one forward pass and one backward pass of all the training examples, in the neural network terminology.

In the paper you mention, they seem to be more flexible regarding the meaning of epoch, as they just define one epoch as being a certain amount of weight updates. You can therefore view one epoch as being an outer loop around the episode loop, as you mentioned in the question.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.