2
$\begingroup$

I am trying to implement an A2C algorithm, but for some reasons, my agent does not learn very well.

I build a custom environment using Unity ML Agents. The environment is very simple: an agent can control a car in a top-down fashion. Here is an example of me playing that environment:

enter image description here

An agent can take 3 actions: turn left, turn right or do not turn at all. An agent can not control the throttle.

On each frame an agent receives a reward equals to a distance traveled along the center-line of the road. Episode finishes if the center of the car is off the road, in this case the agent receives a -1 reward.

To give an agent a sense of motion a single transition consist of 5 frames stacked together. Here is a sample of a single transition:

enter image description here

I used tensorflow and here is how I implemented the computation:

X = tf.placeholder(tf.float32, [None, 128, 128, 3])
A = tf.placeholder(tf.int32, [None])
ADV = tf.placeholder(tf.float32, [None])
R = tf.placeholder(tf.float32, [None])

h1 = conv(X, 32, 8, 4)
pool1 = maxpool(h1, 2, 2)
h2 = conv(pool1, 64, 4, 2)
pool2 = maxpool(h2, 2, 2)
h3 = conv(pool2, 64, 3, 1)
h3 = tf.layers.flatten(h3)
h4 = fc(h3, 512)
actor = fc(h4, NUMBER_OF_ACTIONS, act=None)
critic = fc(h4, 1, act=None)

v0 = tf.squeeze(critic)

prob = tf.nn.softmax(actor)
dist = tf.distributions.Categorical(logits=prob)
a0 = tf.squeeze(dist.sample())

value_loss = tf.reduce_mean(tf.square(tf.squeeze(critic) - R))
action_one_hot = tf.one_hot(A, NUMBER_OF_ACTIONS, dtype=tf.float32)
neg_log_prob = -tf.log(prob)
policy_loss = tf.reduce_mean(
    tf.reduce_sum(neg_log_prob * action_one_hot, axis=1) * ADV)

entropy = tf.reduce_mean(tf.reduce_sum(prob * neg_log_prob, axis=1))
loss = policy_loss + value_loss * VALUE_LOSS_K - entropy * ENTROPY_K
adam = tf.train.AdamOptimizer(LR).minimize(loss)

I trained an agent for a 20 hours on google cloud using GPU and apparently it does not make much progress.

In terms of hyperparameters, I am using learning rate ${\alpha} = 0.0001$ and discount factor ${\gamma} = 0.999$

Some metrics, which I log:

enter image description here

enter image description here

enter image description here

enter image description here

Here is the highest reward episode an agent had during the training (it looks rather disappointing):

enter image description here

I am not sure what I did wrong and thus it is hard to frame the question. Is there something wrong with my approach or implementation ? Any help or suggestions would be greatly appreciated.

Improve this question
$\endgroup$
7
  • $\begingroup$ One thing that immediately comes to my mind is the following: In karpathy.github.io/2016/05/31/rl they do something similar to what I understood you are trying. However, there is one big difference: they give the model the 'sense of motion' by only using two frames and then feeding the difference of these two frames to the NN, so maybe your model does not detect the relationship between these five frames...? I would also try it with the difference of just two frames... $\endgroup$
    – Fabian Werner
    Commented Jun 20, 2019 at 9:08
  • $\begingroup$ Also, do you reward the agent 0 if it stays on track? That could also be a problem because then the NN will never go into the direction of „staying on track“ but rather it will only move away from „leaving the track“ (an event mich more rare in comparison to staying on track)... edit: wait, you say you reward it with the difference between its position and the middle of the road? That means it gets rewarded more if it does not go in the middle? Am I misunderstanding? $\endgroup$
    – Fabian Werner
    Commented Jun 20, 2019 at 13:15
  • $\begingroup$ @FabianWerner, reward is based on the distance an agent traveled along the center-line of the road on each frame. So, if on a particular frame an agent traveled 3 units of distance along the center-line of the road, it would receive 3 points of reward. If an agent traveled 3 units, but 30 degree from the center-line of the road, it would receive $3*cos(30)$ points of reward. Let me know if that makes sense. I will try your suggestion to feed only difference of 2 frames to the model. How do I a calculate the difference between two frames ? Just point-wise subtraction between of pixel values? $\endgroup$
    – koryakinp
    Commented Jun 20, 2019 at 14:10
  • $\begingroup$ Difference: Yes, as far as I understood they just do it pointwise... Concerning the reward: Maybe you have to play around with the reward... Since the model does not seem to work like you did it right now (i.e. the gradient could simply be too small) you could try to make the reward just bigger (multiply it by 2 or 10 or something). Also I did not fully understand how you reward but it should be like this: if it is close to the center line then reward should be extremely high and if it moves away then reward should go down and maybe it should become (very?) negative even though the car is ... $\endgroup$
    – Fabian Werner
    Commented Jun 21, 2019 at 6:22
  • $\begingroup$ still on the road but far away from the center. Thing is: I dont have a clear picture of how the differences will look like and whether they will allow the (C)NN to extract 'I am far away from the center line' easily... maybe you should check that manually. $\endgroup$
    – Fabian Werner
    Commented Jun 21, 2019 at 6:23

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.