I'm trying to find a policy for a simple game using R-learning algorithm. I have a field with values (agent can move in 4 directions) and the goal is to get from starting point to finish point with the highest score.
Final policy gives me incorrect result which doesn't do a right thing, so something definitely wrong with my code/assumplions.
Here's my implementation
def r_learning(game: Game):
states_space_size = 16
actions_space_size = 4
rho = 0
alpha = 0.9 # learning rate for rho value
rsa = np.zeros(shape=(states_space_size, actions_space_size))
beta = 0.9 # learning rate for rsa
max_iterations = 100
s = 0 # initial state; is starting state better?
for i in range(max_iterations):
a = choose_an_action(actions_space_size) # random action selection
r_imm, s_ = perform_action(s, a, game)
urs = get_u_r(s, rsa)
urs_ = get_u_r(s_, rsa)
if random.random() < beta:
rsa[s][a] = r_imm - rho + urs_
# action agrees with a policy?
if random.random() < alpha and rsa[s][a] == urs:
rho = r_imm + urs_ - urs
# change state
s = s_
print(rsa)
return rsa
I've limited number of iterations but what's the actual criterion to stop iterations?
Also I have some questions to clarify:
def get_u_r(state: int, rsa):
return np.max(rsa[state])
for U_R(s) is it sufficient to just select max value from the corresponding R(s, a) matrix row like in the code above?
Should I choose starting state corresponding to my starting point? (I don't think so, because eventually algorithm should fill all the table cells according to the best policy)
Related: Similar question for the same game with Q-learning
Link to full source code: github repo