Q-Learning with mostly 0 reward

Question

I'm implementing Q-learning for an RL problem, but most of the rewards are 0 (with the exception of a +1/-1 at the end of the episode). Does Q-learning still make sense in this context?

I'm also curious as to why people use Q-learning in the first place, instead of simply computing the actual discounted reward for each state and targeting the function approximator at that instead. If you're learning happens offline, you have the entire series of state-action-reward triplets, so computing the discounted reward should be possible (albeit slower).

Answer 1

I'm implementing Q-learning for an RL problem, but most of the rewards are 0 (with the exception of a +1/-1 at the end of the episode). Does Q-learning still make sense in this context?

Certainly. AlphaGo only assigns reward at the end of the episode (and only updates one value per episode). If you want to get more out of fewer episodes, you could use traces.

I'm also curious as to why people use Q-learning in the first place, instead of simply computing the actual discounted reward for each state and targeting the function approximator at that instead.

Computing discounted rewards is the whole point of reinforcement learning.

If you're learning happens offline, you have the entire series of state-action-reward triplets, so computing the discounted reward should be possible (albeit slower).

Q-learning is an online learning algorithm. If all your learning happens offline, then it might not be the best choice.