# How to calculate $max_{\alpha}Q(S',a')$ in q learning?

This is the pseudo code I am using for my reinforcement learning programme.

and consider this is my reward and action-value matrix (these are sample matrices)

R= Reward matrix with row (1:4) as state and column as action(1:4)

[0,0,0,0]
[0,0,0,1]
[0,1,0,0]
[0,1,0,0]

Q= action-value matrix with row (1:4) as state and column as action(1:4)

[2,0,3,0]
[3,4,0,1]
[7,6,8,0]
[0,1,3,0]


where first row(1) is the starting state and last row(4) is the terminal state

if I take gamma as 0.9 and alpha as 0.1 then I can find action-value for Q(3,2) as

$Q(3,2) \leftarrow Q(3,2)+0.1 \times [R(3,2)+0.9\times \max_{0.1}Q(s',a')-Q(3,2)] \\ \\ \hspace{1.8cm} \leftarrow 6 + 0.1 \times [1+0.9\times \max_{0.1}Q(s',a')-6]$

How should I find $max_{0.1}Q(s',a')$?

There should be an "a", not an "alpha" in the subscript, like $max_{a'}{Q(s',a')}$, meaning that you are choosing the maximum value over all the actions for state $s'$. It's just the maximum entry from the corresponding row of $Q$.
• If I understand you correct for Q(3,2) its the maximum value of 3th row, that is $max(7,6,8,0)=8$. then equation becomes $6+0.1*[1+0.9*8-6]=6,22$Am i right?
• That's right. The optimum strategy for this state was to pick the third action, so you adjust the expected value upwards. Note that $s'$ can be different from $s$, depending on which state your action leads you to. Commented Oct 6, 2016 at 15:16
• @Eka NB what Don said about $s'$ being different from $s$: You take the max over the row corresponding to the state you wind up in, not the state you start in. Commented Oct 6, 2016 at 15:19