# Does episodic reinforcement learning still need a discount factor?

The discount factor in reinforcement learning is used to determine how much an agent's decision should be influenced by rewards in the distant future, compared with rewards in the near future. My understanding is that there are two main reasons for this. First, is that with rewards in the distant future, there is greater uncertainty about whether those rewards will actually be acquired by the agent, since there are many steps between the current state and that future state. Second, is that the discount factor ensures that the Bellman equation will converge, since without the discount factor, there is a potentially infinite sum of positive numbers, and introducing the discount factor ensures that these numbers themselves become very small as this sum rolls out.

However, I am wondering if all of this changes when we are dealing with episodic reward learning, where the agent executes actions for a fixed number of steps, and then returns to its initial state. In this case, only the second reason seems to be valid, because there is no longer an infinite sum: the sum is bounded by the length of the episode.

So, in episodic reinforcement learning, should we still be using discount factor due to the first reason I stated above? Or can we ignore the discount factor entirely here?

• "Episodic" does not imply a fixed number of steps. Could you clarify whether you want to ask only about RL with fixed length episodes, or about episodic problems in general? Sep 24, 2019 at 14:14

In episodic settings, you can set $$\gamma = 1$$, since an episode is certain to terminate in finite time. This generally happens because episodic MDPs have "absorbing states", i.e., states for which the system gets stuck and remains there forever. In other words, an episode terminates when an absorbing state is reached. The optimal value of an absorbing states is a known constant, for example, $$v(j) = 0$$, if $$j$$ in an absorbing state. This acts as a boundary condition in the optimality equations, so that they can be solved.