What are the tradeoffs of choosing larger/smaller gamma when performing Value Iteration for MDPs? Will different values of gamma result in different policies?


Different values of gamma may produce different policies. Lower gamma values will put more weight on short-term gains, whereas higher gamma values will put more weight towards long-term gains. Asymptotically, the closer gamma is to 1, the closer the policy will be to one that optimizes the gains over infinite time. On the other hand, value iteration will be slower to converge.

The best gamma depends on your domain. Sometimes it makes sense to look for short term gains (e.g. money gained sooner is actually more valuable than the same amount earned later), other times you want to look as far ahead as you can. And i would say that for a given MDP, there is probably a point (for high values of gamma) where the optimal policies will stabilize (no longer change when you increase gamma even more).

  • 1
    $\begingroup$ In terms of utility, i think this means that the delta U(s) effectively becomes 0, this will depend on the number of timesteps, as gamma gets smaller, the delta =0 is further out. So if you give a static number of steps say 10,000,000 for smaller gamma's the optimal policy may be found at step 100, but at gamma .99 its closer to 1,000,000, at .999 maybe thats the 10,000,000. $\endgroup$ – onaclov2000 Nov 23 '15 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.