0
$\begingroup$

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym.

Copied some code from GitHub which isn't deep yet:

def play_one_game(bins, Q, eps=0.5):
     observation = env.reset()
     done = False
     cnt = 0 # number of moves in an episode
     state = get_state_as_string(assign_bins(observation, bins))
     total_reward = 0

     while not done:
 #        env.render()
         cnt += 1
         # np.random.randn() seems to yield a random action 50% of the time ?
         if np.random.uniform() < eps:
                 act = env.action_space.sample() # epsilon greedy
         else:
                 act = max_dict(Q[state])[0]

         observation, reward, done, _ = env.step(act)

         total_reward += reward
         #[1]punish positions far away from 0
         #[1]total_reward -= abs(observation[0])*0.3

         if done and cnt < 200:
                 reward = -300

         state_new = get_state_as_string(assign_bins(observation, bins))

         a1, max_q_s1a1 = max_dict(Q[state_new])
         Q[state][act] += ALPHA*(reward + GAMMA*max_q_s1a1 - Q[state][act])
         state, act = state_new, a1

     return total_reward, cnt

The example worked, but in my opinion it was more like cheating... The CartPole stood upright for the 200 step which are demanded. However the cart moved heavily and drifted away from the starting point. To stop this I tried to penalize the distance from the starting point with those two lines marked with [1], and gave it some more training time (about 30000 Iterations instead of the 10000 before).

The result weren't as good as expected. It was worse than before even if it didn't failed completely. However it missed my goal to stay centered.

Did I gave it to few time to train, since I've made a more complicated model? Or was this generally a bad idea of penalizing?

$\endgroup$
  • 1
    $\begingroup$ Have you tried various weights of that penalization? $\endgroup$ – Karel Macek Nov 1 '18 at 6:36
  • $\begingroup$ Yes. I tried many weights between 0 and 20. Everything bigger 2 was somehow obviously wrong. Complete escalation. Weights between 1.5 and 1.9 seemed to stabalize after time. However it didn‘t effect the distance at all. It got even worse than without this penalty. $\endgroup$ – Mr.Sh4nnon Nov 1 '18 at 6:50
  • $\begingroup$ How does the resulting policy look like? Isn't all about one action (no changes)? $\endgroup$ – Karel Macek Nov 1 '18 at 7:33
  • $\begingroup$ I‘m not sure if I understand right. Since I didn’t changed the formula for the Q value, the policy has to be the same I guess. I only added a weighted term of distance to change the given reward. $\endgroup$ – Mr.Sh4nnon Nov 1 '18 at 7:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.