11
$\begingroup$

I am trying to use a Neural Network in order to approximate the Q-value in Q-learning as in Questions about Q-Learning using Neural Networks. As suggested in the first answer, I am using a linear activation function for the output layer, while I am still using the sigmoid activation function in the hidden layers (2, although I can change this later on). I am also using a single NN that returns an output for each action $Q(a)$ as advised.

However, the algorithm is still diverging for the simple cart-pole balancing problem. So, I fear my Q-update is wrong. After the initialization, what I have done at each step is the following:

  • Calculate $Q_t(s_t)$ using forward propagation of the NN for all actions.
  • Select a new action, $a_t$, land in a new state $s_t$.
  • Calculate $Q_t(s_{t+1})$ using forward propagation of the NN for all actions.
  • Set the target Q-value as: $Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t \left[r_{t+1}+\gamma \max_a Q(s_{t+1},a) - Q_t(s_t,a_t) \right]$ only for the current action, $a_t$, whilst setting $Q_{t+1}(s,a_t)=Q_{t}(s,a_t)$ for the other states. Note, I think this is the problem.
  • Set the error vector to $\mathbf{e}=Q_\mathrm{target}-Q_t=Q_{t+1}-Q_t$
  • Backpropagate the error through the NN in order to update the weight matrices.

Could anyone please point out to me where I have gone wrong?

Besides, do you reckon I should include a bias term as well in the input layer and the first hidden layer (i.e. for the sigmoid functions)? Will it make a difference?

Thank you very much in advance for your help. I can help clarify the question or share code if required.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
10
$\begingroup$

Your target should be just

$r_{t+1}+\gamma \max_a Q(s_{t+1},a)$.

Note that your error term (which is correct) could then be rewritten as $r_{t+1}+\gamma \max_a Q(s_{t+1},a) - Q_t$

which is the term inside brackets in the update formula. This will get multiplied by your NN learning rate and other backpropagation terms during learning, and then added to the previous weights, just like the $Q$ update formula.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you for the tip! That way I can avoid numerical errors in the target value: as you say, it is pointless to add and then subtract $Q_t$. $\endgroup$
    – Enrico Anderlini
    Commented Mar 9, 2016 at 9:57
  • $\begingroup$ However, my NN+Q-learning is still not converging. So, if the method is correct, I guess I will just try to play around with the learning rate, and number of neurons in the hidden layer(s) $\endgroup$
    – Enrico Anderlini
    Commented Mar 9, 2016 at 9:58
  • 1
    $\begingroup$ Yes, you should do that (try to use the ReLU transfer function too). But note that Q-Learning has no convergence guarantees with function approximation. If your next try does not work, you may want to try Advantage Learning. And finally, the problem might be that your data is not iid. So you'd need to implement Experience Replay. $\endgroup$
    – rcpinto
    Commented Mar 9, 2016 at 16:03
  • $\begingroup$ I have an extension to the question. I understand the algorithm as mentioned in the question but am confused about the $Q_{t+1}$ and the $r_{t+1}$ terms. If the reward for the game is known only at the end of the game (at a win or a loss condition), how are we to provide the reward for an intermediate step? $\endgroup$
    – Kanishka Ganguly
    Commented Nov 26, 2016 at 5:00
  • 1
    $\begingroup$ You don't, it is zero. But it will be slowly propagated by the Q values to previous states, because the Q value is an estimate of cumulative future rewards. $\endgroup$
    – rcpinto
    Commented Nov 26, 2016 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.