This is the pseudo code I am using for my reinforcement learning programme.

enter image description here

and consider this is my reward and action-value matrix (these are sample matrices)

R= Reward matrix with row (1:4) as state and column as action(1:4)


Q= action-value matrix with row (1:4) as state and column as action(1:4)


where first row(1) is the starting state and last row(4) is the terminal state

if I take gamma as 0.9 and alpha as 0.1 then I can find action-value for Q(3,2) as

$Q(3,2) \leftarrow Q(3,2)+0.1 \times [R(3,2)+0.9\times \max_{0.1}Q(s',a')-Q(3,2)] \\ \\ \hspace{1.8cm} \leftarrow 6 + 0.1 \times [1+0.9\times \max_{0.1}Q(s',a')-6]$

How should I find $max_{0.1}Q(s',a')$?


1 Answer 1


There should be an "a", not an "alpha" in the subscript, like $max_{a'}{Q(s',a')}$, meaning that you are choosing the maximum value over all the actions for state $s'$. It's just the maximum entry from the corresponding row of $Q$.

  • $\begingroup$ If I understand you correct for Q(3,2) its the maximum value of 3th row, that is $max(7,6,8,0)=8$. then equation becomes $6+0.1*[1+0.9*8-6]=6,22$Am i right? $\endgroup$
    – Eka
    Commented Oct 6, 2016 at 14:23
  • 1
    $\begingroup$ That's right. The optimum strategy for this state was to pick the third action, so you adjust the expected value upwards. Note that $s'$ can be different from $s$, depending on which state your action leads you to. $\endgroup$
    – Don Reba
    Commented Oct 6, 2016 at 15:16
  • 1
    $\begingroup$ @Eka NB what Don said about $s'$ being different from $s$: You take the max over the row corresponding to the state you wind up in, not the state you start in. $\endgroup$ Commented Oct 6, 2016 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.