# Convergence criterion for R-learning algorithm

I'm trying to find a policy for a simple game using R-learning algorithm. I have a field with values (agent can move in 4 directions) and the goal is to get from starting point to finish point with the highest score.

Final policy gives me incorrect result which doesn't do a right thing, so something definitely wrong with my code/assumplions.

Here's my implementation

def r_learning(game: Game):
states_space_size = 16
actions_space_size = 4
rho = 0
alpha = 0.9  # learning rate for rho value
rsa = np.zeros(shape=(states_space_size, actions_space_size))
beta = 0.9  # learning rate for rsa
max_iterations = 100
s = 0  # initial state; is starting state better?
for i in range(max_iterations):
a = choose_an_action(actions_space_size)  # random action selection
r_imm, s_ = perform_action(s, a, game)
urs = get_u_r(s, rsa)
urs_ = get_u_r(s_, rsa)
if random.random() < beta:
rsa[s][a] = r_imm - rho + urs_

# action agrees with a policy?
if random.random() < alpha and rsa[s][a] == urs:
rho = r_imm + urs_ - urs

# change state
s = s_
print(rsa)
return rsa


I've limited number of iterations but what's the actual criterion to stop iterations?

Also I have some questions to clarify:

def get_u_r(state: int, rsa):
return np.max(rsa[state])

1. for U_R(s) is it sufficient to just select max value from the corresponding R(s, a) matrix row like in the code above?

2. Should I choose starting state corresponding to my starting point? (I don't think so, because eventually algorithm should fill all the table cells according to the best policy)

Link to full source code: github repo