2
$\begingroup$

I am implementing some of the basic reinforcement learning algorithms but ran into a problem with an online (one-step TD) A2C implementation where my reward seems to decrease over time instead of increasing (light blue in below picture).

I already implemented REINFORCE (offline, dark blue) and a Q-Actor-Critic (online with one-step TD, red) algorithm that both seem to work fine (no baselines, though).

The mean_episode_reward is calculated by running the policy 100 times in a simple, static 5x5 grid world with two terminal states that reward +1 or -1 and a discount of 0.99

mean episode loss with 100 episodes for three algorithm

I used a single-layer feed-forward neural network for REINFORCE and Q-AC for the policy and added a second head for A2C, now:

neural network structure

The training cycle is quite simple, where I get predictions for each action and the state-value from the model (for a state s), pick an action, collect a reward and get new action- and state-value-predictions. These are then used to calculate the advantage (one-step TD), i.e. $advantage = reward + discount * v(s') - v(s)$. This is then used to calculate the loss for the actor and critic which is then added (actor_loss + 0.5 * critic_loss) and fed to the gradient tape:

done = False

# initial
prediction, v_s = self.model(self.input_fn(), training=True)

while not done:
  with tf.GradientTape() as tape:
    action = self.chooseAction(prediction) # [0, 1, 2, 3] for up, right, ...
    reward, done = self.env.step(self.actions[int(action)])

    prediction, v_s_prime = self.model(self.input_fn(), training=True)
    advantage = reward + self.discount * v_s_prime - v_s

    actor_loss = tf.squeeze(self.actor_loss(action, prediction, advantage))
    critic_loss = tf.squeeze(self.critic_loss(reward + self.discount * v_s_prime, v_s))
    loss = actor_loss + 0.5 * critic_loss

  gradient = tape.gradient(loss, self.model.trainable_variables)
  self.optimizer.apply_gradients(zip(gradient, self.model.trainable_variables))
  v_s = v_s_prime

I don't know where I'm stuck as I would expect my implementation to maximise the reward, not minimise. I would highly appreciate any hint.

Edit: yes, when multiplying with the negative advantage the performance does improve - it's just not what I expected to happen.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.