1
$\begingroup$

Let's imagine a Q-learning version of the supervised learning problem of guessing a digit from MNIST database. In this game, the initial state is the 28x28 image pixels. You have 10 possible actions, labeled from 0 to 9. And the reward you get is $R$ if you correctly guessed the digit, 0 otherwise. After each image/guess, the game ends.

As the second state is always terminal and assuming $\alpha = 1$, I simply update my $Q(s,a)$ with $Q(s,a) \leftarrow R$.

I noticed I get some very different learning curves when I change $R$, which I didn't expect. The model converges way faster when $R$ is big.

My guess is it depends of the magnitude of $Q(s,a)$ values. If $Q(s,a)$ are 3 digit numbers, using $R = 1$ would be too "low" and would be equivalent to a 0. However, using $R = 100000$ would be so "big" compared to 3 digit numbers that the updated $Q(s,a)$ would be similar to one-hot encoded vector.

Is that a common issue?

$\endgroup$

2 Answers 2

1
$\begingroup$

Neural networks are sensitive to the scale of the input and the target. Rescaling the target implicitly requires rescaling the network weights, which in turn will influence the ability of the optimizer to find a good solution.

$\endgroup$
0
$\begingroup$

Since you can do only one action before the game ends, this looks like a k-armed bandit problem. The problem is that, unless you are using some kind of Q function approximator, like a neural network, the agent will not be able to understand from a certain reward how to "classify" (or, more properly here, what action to execute) similar digits from the input image/state, i.e. it doesn't learn how to generalize to different images of the same digit. So, since te reward for a (state,action) couple is deterministic, the agent should just try all possible combinations to really understand how to act.

So, as I said, the agent in the end woukd just have a Q table that has as many entries as the state/actions combinations, so the number of images*10, with a single reward for each action relative to a state (so nine 0s and one 1 for each state). Since the final policy should be to choose the highest rewarding action from each state based on the Q table, I don't see how a bigger R could make a difference, but maybe it could be due to initialization factors. Try to explain in more detail your algorithm for a more accurate answer.

$\endgroup$
2
  • $\begingroup$ I don't use a Q-table, but a neural network. My issue comes from the outputs (logits) of the network, that have too large values compared to a reward of 1 $\endgroup$
    – bill0ute
    Mar 7, 2017 at 16:52
  • 1
    $\begingroup$ In this case, the problem is probably that you network is non converging to a solution, but diverging. When training a NN, amplitudes matter, this is why you need to tweak hyperparameters of the net, like the training step value, usually called alpha. Try to tweak its parameters, setting a lower training step. $\endgroup$
    – dante
    Mar 9, 2017 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.