I'm trying to solve the same problem with different algorithms (Travel max possible distance with a car). While using value iteration and policy iteration I was able to get the best results possible but with Q-learning it doesn't seem to go well. My algorithm looks like this
alpha = 0.2 # learning rate
gamma = 0.9 # discount factor
Q = np.zeros(shape=(70, 7)) # tank size X steps, actions space size
theta = 0.01 # track Q-function updates
# while Q is not converged
for i in count():
delta = 0 # max Q update for the episode
s = 0 # starting state
while True: # running an episode
# exploration vs exploitation
a = get_next_action(Q, s, eps=eps) # random or follow policy
# perform action chosen, receive reward and new state
r, s_ = get_action_reward(a, s)
max_q = maximize_Q(s_) # maximum Q value we can get from state s_
q_update = alpha*(r + gamma*max_q - Q[s, a])
Q[s, a] += q_update
s = s_ # do to next state
delta = max(delta, np.abs(q_update))
if s is None: # while s is not terminal
break
# check Q for convergence
if delta < theta:
print(f'Q-function converged after {i} iterations')
break
# exploit more with each iteration
eps = min_eps + (max_eps - min_eps)*np.exp(-decay_rate*i)
My reward is distance_travelled
or -step_number
(if we did not move).
Available actions are
ACTIONS = (
(30, 0), # buy 30L without driving
(20, 0),
(10, 0),
(0, 0), # have some rest
(-10, 10),
(-20, 20),
(-30, 30), # burn 30L of fuel to drive 30 km
)
And the size of a tank is limited, so you cannot have more than 60L in it.
What might be wrong with Q-algorithm itself or its hyperparameters, so it does not give me 150km
result which is the best for 10
timesteps