# Q-learning with Neural Network as function approximation

I am trying to use a Neural Network in order to approximate the Q-value in Q-learning as in Questions about Q-Learning using Neural Networks. As suggested in the first answer, I am using a linear activation function for the output layer, while I am still using the sigmoid activation function in the hidden layers (2, although I can change this later on). I am also using a single NN that returns an output for each action $Q(a)$ as advised.

However, the algorithm is still diverging for the simple cart-pole balancing problem. So, I fear my Q-update is wrong. After the initialization, what I have done at each step is the following:

• Calculate $Q_t(s_t)$ using forward propagation of the NN for all actions.
• Select a new action, $a_t$, land in a new state $s_t$.
• Calculate $Q_t(s_{t+1})$ using forward propagation of the NN for all actions.
• Set the target Q-value as: $Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t \left[r_{t+1}+\gamma \max_a Q(s_{t+1},a) - Q_t(s_t,a_t) \right]$ only for the current action, $a_t$, whilst setting $Q_{t+1}(s,a_t)=Q_{t}(s,a_t)$ for the other states. Note, I think this is the problem.
• Set the error vector to $\mathbf{e}=Q_\mathrm{target}-Q_t=Q_{t+1}-Q_t$
• Backpropagate the error through the NN in order to update the weight matrices.

Could anyone please point out to me where I have gone wrong?

Besides, do you reckon I should include a bias term as well in the input layer and the first hidden layer (i.e. for the sigmoid functions)? Will it make a difference?

Thank you very much in advance for your help. I can help clarify the question or share code if required.

$r_{t+1}+\gamma \max_a Q(s_{t+1},a)$.
Note that your error term (which is correct) could then be rewritten as $r_{t+1}+\gamma \max_a Q(s_{t+1},a) - Q_t$
which is the term inside brackets in the update formula. This will get multiplied by your NN learning rate and other backpropagation terms during learning, and then added to the previous weights, just like the $Q$ update formula.
• Thank you for the tip! That way I can avoid numerical errors in the target value: as you say, it is pointless to add and then subtract $Q_t$. – Enrico Anderlini Mar 9 '16 at 9:57
• I have an extension to the question. I understand the algorithm as mentioned in the question but am confused about the $Q_{t+1}$ and the $r_{t+1}$ terms. If the reward for the game is known only at the end of the game (at a win or a loss condition), how are we to provide the reward for an intermediate step? – Kanishka Ganguly Nov 26 '16 at 5:00