0
$\begingroup$

Maybe this is better posted on stackoverflow, but I thought I would start here . I am trying to understand how RL policy gradient style algorithms are implemented in autodiff frameworks like TensorFlow/Keras.

Here is an implementation of sample policy gradient or A2C in the Keras framework, See here as an example of a full working code repo for the above snippet:

https://gist.github.com/kkweon/c8d1caabaf7b43317bc8825c226045d2

in lines 78-88 they use the following code to update the policy:

    action_prob_placeholder = self.model.output
    action_onehot_placeholder = K.placeholder(shape=(None, self.output_dim),
                                              name="action_onehot")
    discount_reward_placeholder = K.placeholder(shape=(None,),
                                                name="discount_reward")

    **action_prob = K.sum(action_prob_placeholder * action_onehot_placeholder, axis=1)**
    log_action_prob = K.log(action_prob)

    loss = - log_action_prob * discount_reward_placeholder
    loss = K.mean(loss)

I am struggling to understand how the

K.sum(action_prob_placeholder * action_onehot_placeholder, axis=1)

term above works, specifically, how is $\pi(a_t|s_t)$ is expressed as

action_prob_placeholder * action_onehot_placeholder

Any explanations would be appreciated. thank you

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.