I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym.
Copied some code from GitHub which isn't deep yet:
def play_one_game(bins, Q, eps=0.5): observation = env.reset() done = False cnt = 0 # number of moves in an episode state = get_state_as_string(assign_bins(observation, bins)) total_reward = 0 while not done: # env.render() cnt += 1 # np.random.randn() seems to yield a random action 50% of the time ? if np.random.uniform() < eps: act = env.action_space.sample() # epsilon greedy else: act = max_dict(Q[state]) observation, reward, done, _ = env.step(act) total_reward += reward #punish positions far away from 0 #total_reward -= abs(observation)*0.3 if done and cnt < 200: reward = -300 state_new = get_state_as_string(assign_bins(observation, bins)) a1, max_q_s1a1 = max_dict(Q[state_new]) Q[state][act] += ALPHA*(reward + GAMMA*max_q_s1a1 - Q[state][act]) state, act = state_new, a1 return total_reward, cnt
The example worked, but in my opinion it was more like cheating... The CartPole stood upright for the 200 step which are demanded. However the cart moved heavily and drifted away from the starting point. To stop this I tried to penalize the distance from the starting point with those two lines marked with , and gave it some more training time (about 30000 Iterations instead of the 10000 before).
The result weren't as good as expected. It was worse than before even if it didn't failed completely. However it missed my goal to stay centered.
Did I gave it to few time to train, since I've made a more complicated model? Or was this generally a bad idea of penalizing?