I am reading the chapter 3 of Sutton's book Reinforcement Learning: An Introduction and I am confused by episodic task and continuous tasks.
In reinforcement learning, how to distinguish episodic task and continuous tasks?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
A continuous task never ends. Which means you're not given the reward at the end, since there is no end, but every so often during the task.
For example, reading the internet to learn maths could be considered a continuous task.
An episodic task lasts a finite amount of time. For example, playing a single game of Go is an episodic task, which you win or lose. In an episodic task, there might be only a single reward, at the end of the task, and one option is to distribute the reward evenly across all actions taken in that episode.
In a continuous task, rewards might be assigned with discounting, so more recent reactions receive greater reward, and actions a long time in the past receive a vanishingly small reward. For example the reward could be geometric with distance in the past, with a discount factor $\lambda \in [0, 1]$.
answered Apr 2, 2017 at 8:50
-
$\begingroup$ But in the book, “Intro to Reinforcement learning” written by Sutton, it said "Pole balancing can be interpreted in two ways, one is continuing task and the other is episodic task”. But I can't understand it why it can be treated as continuing task because it has a obvious terminate state when falling off the poll… $\endgroup$ Commented Oct 19, 2018 at 7:53
-
2$\begingroup$ No, in case of the continuing formulation, the task would continue after a reset. For example, let's say we start at
t=0and att=kthere is the first failure. Now, the task would continue and at timet=k+1the pole would be in its starting state. So by timestept=100,000we might have had 1000 failures but this is still the first and only 'episode'. $\endgroup$– anaikCommented Jan 2, 2019 at 6:00
$\begingroup$
$\endgroup$
A continuous task can go on forever, an episodic task has at least one finite state (i.e. an end of the game). Mathematically speaking an episodic task has a state with transition probability 1 to itself and 0 anywhere else.
This is highly interpretable, as gets clear in question 3.7 of the book: Is a maze (with reward 1 for escaping, else reward 0) episodic? Technically yes, there is a final state but theoretically it can still go on forever (running in circles). Treating this as episodic to train an agent will lead to him never reaching the end in a fixed amount of rounds per game and fixed amount of games, therefore not training at all.
