# When to use low discount factor in reinforcement learning?

In reinforcement learning, we're trying to maximize long-term rewards weighted by a discount factor $$\gamma$$: $$\sum_{t=0}^\infty \gamma^t r_t$$.

$$\gamma$$ is in the range $$[0,1]$$, where $$\gamma=1$$ means a reward in the future is as important as a reward on the next time step and $$\gamma=0$$ means that only the reward on the next time step is important. Formally, $$\gamma$$ is given as part of the problem, but this isn't the case in practice where choices must be made on how to build the states, actions, and rewards of the MDP out of real world information.

In my experience (which is far from comprehensive), the value of $$\gamma$$ used is typically high, such as 0.9 or 0.99 or 0.999. (Or simply 1.0 if we are restricted by a finite time horizon.) But this seems mostly arbitrary.

My question is: when might we use a low, but non-zero value for $$\gamma$$, such as 0.5 or 0.1?

I'm asking mostly out of curiosity, the question occurred to me and I thought I'd see whether any of you had seen something like this before.

The intuitive answer would be that $$\gamma$$ is low when the immediate rewards are much more important than long-term rewards, but that's strange. What environment could you be in where you still care about the future, but not that much? What kind of policy would you learn in an environment like that?

The discount factor doesn't really have a well founded interpretation as far as I know. It seems to have been introduced primarily so that the problem is more mathematically or computationally well-behaved. People have interpreted it as a "life-span risk" factor, (i.e $$\gamma$$ is your chance of dying each time-step, so you should weigh anticipated future reward accordingly). Personally I don't really buy it, because this could just be easily built in to the environment itself. Another interpretation is that it mimics human time preferences, but I don't really buy this either -- the point of reinforcement learning isn't really to mimic human behavior. You can see a bit more discussion on these points in the introduction here.