Convergence criterion for R-learning algorithm

Question

I'm trying to find a policy for a simple game using R-learning algorithm. I have a field with values (agent can move in 4 directions) and the goal is to get from starting point to finish point with the highest score.

Final policy gives me incorrect result which doesn't do a right thing, so something definitely wrong with my code/assumplions.

Here's my implementation

def r_learning(game: Game):
    states_space_size = 16
    actions_space_size = 4
    rho = 0
    alpha = 0.9  # learning rate for rho value
    rsa = np.zeros(shape=(states_space_size, actions_space_size))
    beta = 0.9  # learning rate for rsa
    max_iterations = 100
    s = 0  # initial state; is starting state better?
    for i in range(max_iterations):
        a = choose_an_action(actions_space_size)  # random action selection
        r_imm, s_ = perform_action(s, a, game)
        urs = get_u_r(s, rsa)
        urs_ = get_u_r(s_, rsa)
        if random.random() < beta:
            rsa[s][a] = r_imm - rho + urs_

        # action agrees with a policy?
        if random.random() < alpha and rsa[s][a] == urs:
            rho = r_imm + urs_ - urs

        # change state
        s = s_
    print(rsa)
    return rsa

I've limited number of iterations but what's the actual criterion to stop iterations?

Also I have some questions to clarify:

def get_u_r(state: int, rsa):
    return np.max(rsa[state])

1. for U_R(s) is it sufficient to just select max value from the corresponding R(s, a) matrix row like in the code above?

2. Should I choose starting state corresponding to my starting point? (I don't think so, because eventually algorithm should fill all the table cells according to the best policy)

---

Related: Similar question for the same game with Q-learning

Link to full source code: github repo