How can I deal with the Reinforcement learning problem with a fixed length episode but no terminal state?

Can I use the same method as the normal reinforcement learning with terminal state?


1 Answer 1


You must define the states to match the learning objective you have set for the agent, and it is entirely possible that $\text{state} \neq \text{current_observation}$, when considering e.g. a map view on a grid world, or sensor readings etc as your $\text{current_observation}$. The state representation can include key metadata such as internal features of the agent, counters of events (such as numbers of times each action taken so far - or even a complete history to date), and current time. You should include any data in the state representation that affects future rewards. This is required to meet the assumption in reinforcement learning that the state has the Markov property.

Ending the episode is the terminal state regardless of representation. For most RL techniques that will mean:

  • You actually stop the episode when running or simulating the environment

  • The expected return is fixed at zero when calculating value functions from that point (this is important for boot-strapping methods such as TD learning)

In your specific case, as you have a fixed length episode:

  • You should include the current time step (or time remaining) as part of the state, because that will heavily influence the return - i.e. the expected sum of all remaining rewards - from any given state.

This last point means effectively you do have a terminal state, it includes all state representations with the time set at the end.

This answer assumes that "a fixed length episode" is a deliberate part of the learning goals for the agent - your plan is that the agent must navigate the environment and collect maximum possible reward in a fixed time. The alternative is that you are limiting training length for some reason, but actually want to train an agent to maximise return in a continuous non-episodic environment - the answer does not address that.

  • $\begingroup$ But in my scenario, just as visual object tracking, there is no terminal state but a time length during which the agent track the target, how should i deal with this ? $\endgroup$
    – mac wang
    Commented Oct 29, 2017 at 5:56
  • $\begingroup$ @macwang: "a time length during which the agent track the target" That defines the terminal state. You have given the agent a time limit. State includes all relevant information that will predict expected return. The time limit is therefore part of the state, as I have written in my answer. "Time is up" equals "terminal state" $\endgroup$ Commented Oct 29, 2017 at 9:25
  • $\begingroup$ @macwang: I have updated the answer to make this clearer. It might be that you don't really intend "a fixed length episode" to be part of the learning objective for the agent? It might be for some other purpose, such as training in different scenarios? In which case please update the question to make this clear, because the word "episode" has very specific meaning in Reinforcement Learning. $\endgroup$ Commented Oct 29, 2017 at 10:08
  • $\begingroup$ Thank you so much. I used to consider the alternative one in the last part of your answer. Is it possible for solving continuous scenario with a fixed-length episode? $\endgroup$
    – mac wang
    Commented Oct 30, 2017 at 23:00
  • $\begingroup$ @macwang: Yes you can train on fixed length parts when solving continuous problems. You need to edit the question with some details of your problem and where you are stuck if this is the case (have you tried anything, what isn't working, or what don't you understand - maybe show the RL algorithm you are using to help explain why you have doubts that it will "just work"), so that the question and answer match. $\endgroup$ Commented Oct 31, 2017 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.