Q-Learning: why changing reward value affects convergence? Let's imagine a Q-learning version of the supervised learning problem of guessing a digit from MNIST database. In this game, the initial state is the 28x28 image pixels. You have 10 possible actions, labeled from 0 to 9. And the reward you get is $R$ if you correctly guessed the digit, 0 otherwise. After each image/guess, the game ends.
As the second state is always terminal and assuming $\alpha = 1$, I simply update my $Q(s,a)$ with $Q(s,a) \leftarrow R$.
I noticed I get some very different learning curves when I change $R$, which I didn't expect. The model converges way faster when $R$ is big.
My guess is it depends of the magnitude of $Q(s,a)$ values. If $Q(s,a)$ are 3 digit numbers, using $R = 1$ would be too "low" and would be equivalent to a 0. However, using $R = 100000$ would be so "big" compared to 3 digit numbers that the updated $Q(s,a)$ would be similar to one-hot encoded vector.
Is that a common issue?  
 A: Neural networks are sensitive to the scale of the input and the target. Rescaling the target implicitly requires rescaling the network weights, which in turn will influence the ability of the optimizer to find a good solution.
A: Since you can do only one action before the game ends, this looks like a k-armed bandit problem. The problem is that, unless you are using some kind of Q function approximator, like a neural network, the agent will not be able to understand from a certain reward how to "classify" (or, more properly here, what action to execute) similar digits from the input image/state, i.e. it doesn't learn how to generalize to different images of the same digit. So, since te reward for a (state,action) couple is deterministic, the agent should just try all possible combinations to really understand how to act. 
So, as I said, the agent in the end woukd just have a Q table that has as many entries as the state/actions combinations, so the number of images*10, with a single reward for each action relative to a state (so nine 0s and one 1 for each state). 
Since the final policy should be to choose the highest rewarding action from each state based on the Q table, I don't see how a bigger R could make a difference, but maybe it could be due to initialization factors. 
Try to explain in more detail your algorithm for a more accurate answer. 
