What are the tradeoffs of choosing larger/smaller gamma when performing Value Iteration for MDPs? Will different values of gamma result in different policies?


Different values of gamma may produce different policies. Lower gamma values will put more weight on short-term gains, whereas higher gamma values will put more weight towards long-term gains. Asymptotically, the closer gamma is to 1, the closer the policy will be to one that optimizes the gains over infinite time. On the other hand, value iteration will be slower to converge.

The best gamma depends on your domain. Sometimes it makes sense to look for short term gains (e.g. money gained sooner is actually more valuable than the same amount earned later), other times you want to look as far ahead as you can. And i would say that for a given MDP, there is probably a point (for high values of gamma) where the optimal policies will stabilize (no longer change when you increase gamma even more).

  • 1
    $\begingroup$ In terms of utility, i think this means that the delta U(s) effectively becomes 0, this will depend on the number of timesteps, as gamma gets smaller, the delta =0 is further out. So if you give a static number of steps say 10,000,000 for smaller gamma's the optimal policy may be found at step 100, but at gamma .99 its closer to 1,000,000, at .999 maybe thats the 10,000,000. $\endgroup$ – onaclov2000 Nov 23 '15 at 0:21

There is already a good answer to this question, but I would like to add something more to it.

The gamma(discounting factor) is a reflection of how you value your future reward.

Choosing the gamma value=0 would mean that you are going for a greedy policy where for the learning agent, what happens in the future does not matter at all. The gamma value of 0 is the best when unit testing the code, as for MDPs, it is always difficult to test the behavior of an agent when number of horizons increases.

On the other hand, choosing the gamma value=1 would mean that you don't have any preference about when you want to get the reward; that is to say, for example, if you are given a reward in time step 3 or in time step 7 would be of equal value to your agent.

These are the edge scenarios, but you would like to have something in between to find an optimal policy. In my several years experience with MDPs, the most common value of gamma I have seen, is 0.9, though the value always depends on the requirements of the problem domain.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.