I am implementing some of the basic reinforcement learning algorithms but ran into a problem with an online (one-step TD) A2C implementation where my reward seems to decrease over time instead of increasing (light blue in below picture).

I already implemented REINFORCE (offline, dark blue) and a Q-Actor-Critic (online with one-step TD, red) algorithm that both seem to work fine (no baselines, though).

The mean_episode_reward is calculated by running the policy 100 times in a simple, static 5x5 grid world with two terminal states that reward +1 or -1 and a discount of 0.99

mean episode loss with 100 episodes for three algorithm

I used a single-layer feed-forward neural network for REINFORCE and Q-AC for the policy and added a second head for A2C, now:

neural network structure

The training cycle is quite simple, where I get predictions for each action and the state-value from the model (for a state s), pick an action, collect a reward and get new action- and state-value-predictions. These are then used to calculate the advantage (one-step TD), i.e. $advantage = reward + discount * v(s') - v(s)$. This is then used to calculate the loss for the actor and critic which is then added (actor_loss + 0.5 * critic_loss) and fed to the gradient tape:

done = False

# initial
prediction, v_s = self.model(self.input_fn(), training=True)

while not done:
  with tf.GradientTape() as tape:
    action = self.chooseAction(prediction) # [0, 1, 2, 3] for up, right, ...
    reward, done = self.env.step(self.actions[int(action)])

    prediction, v_s_prime = self.model(self.input_fn(), training=True)
    advantage = reward + self.discount * v_s_prime - v_s

    actor_loss = tf.squeeze(self.actor_loss(action, prediction, advantage))
    critic_loss = tf.squeeze(self.critic_loss(reward + self.discount * v_s_prime, v_s))
    loss = actor_loss + 0.5 * critic_loss

  gradient = tape.gradient(loss, self.model.trainable_variables)
  self.optimizer.apply_gradients(zip(gradient, self.model.trainable_variables))
  v_s = v_s_prime

I don't know where I'm stuck as I would expect my implementation to maximise the reward, not minimise. I would highly appreciate any hint.

Edit: yes, when multiplying with the negative advantage the performance does improve - it's just not what I expected to happen.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.