Is gamma actually an efficient way to weigh future rewards in reinforcement learning?

Question

Typically the discounted sum of rewards is defined as follows:

G_t = Sum(gamma ** n * reward_t...)

But this means that rewards are worth exponentially less with each timestep. Wouldn't it make more sense to have reward weighting looking something like this?

So we maximise rewards at some t + n in the future instead of trying to maximise rewards at t + 1 and then exponentially decaying the weights thereafter?

What is the intuition or reasoning behind the g ** n weighting of rewards compared to any other function of reward weighting?

Answer 1

Exponential discounting is "time-consistent" in a way that other forms of discounting are not. For example, with $\gamma = 0.9$, you would prefer 1 reward today to 1 reward tomorrow, and 1 reward in 10 days to 1 reward in 11 days. You would also prefer 2 reward tomorrow over 1 reward today, and 2 reward in 11 days over 1 reward in 10 days.

Under your scheme, it seems like you'd prefer 1 reward tomorrow over 1 reward today, but you'd prefer 1 reward in 10 days over 1 reward in 11 days. You might prefer 1 reward today over 2 tomorrow, but 2 in 11 days rather than 1 in 10 days.

So you answer differently to the same questions depending on how far away something is, which is a bit strange. If taking these rewards required longer term planning and preperation, you might find yourself spending a few days to prepare to do X, only to later change your mind and throw it all away to do Y.

Another popular alternative to exponential discounting is hyperbolic discounting, which is supposedly what humans use. However this is also not time-consistent.

Practically speaking, it's a bit nontrivial to use alternate discount functions because the Bellman equation, the basis of many reinforcement learning algorithms, assumes exponential discounting. Fedus et al show you can tweak some things to make hyperbolic discounting work with Q-learning.

Another practical reason for exponential discounting is that it converges, whereas a hyperbolic sum of rewards might diverge to infinity. So it makes things nice for theoretical analysis.