What are the tradeoffs of choosing larger/smaller gamma when performing Value Iteration for MDPs? Will different values of gamma result in different policies?
Different values of gamma may produce different policies. Lower gamma values will put more weight on short-term gains, whereas higher gamma values will put more weight towards long-term gains. Asymptotically, the closer gamma is to 1, the closer the policy will be to one that optimizes the gains over infinite time. On the other hand, value iteration will be slower to converge.
The best gamma depends on your domain. Sometimes it makes sense to look for short term gains (e.g. money gained sooner is actually more valuable than the same amount earned later), other times you want to look as far ahead as you can. And i would say that for a given MDP, there is probably a point (for high values of gamma) where the optimal policies will stabilize (no longer change when you increase gamma even more).