Maybe this is better posted on stackoverflow, but I thought I would start here . I am trying to understand how RL policy gradient style algorithms are implemented in autodiff frameworks like TensorFlow/Keras.

Here is an implementation of sample policy gradient or A2C in the Keras framework, See here as an example of a full working code repo for the above snippet:


in lines 78-88 they use the following code to update the policy:

    action_prob_placeholder = self.model.output
    action_onehot_placeholder = K.placeholder(shape=(None, self.output_dim),
    discount_reward_placeholder = K.placeholder(shape=(None,),

    **action_prob = K.sum(action_prob_placeholder * action_onehot_placeholder, axis=1)**
    log_action_prob = K.log(action_prob)

    loss = - log_action_prob * discount_reward_placeholder
    loss = K.mean(loss)

I am struggling to understand how the

K.sum(action_prob_placeholder * action_onehot_placeholder, axis=1)

term above works, specifically, how is $\pi(a_t|s_t)$ is expressed as

action_prob_placeholder * action_onehot_placeholder

Any explanations would be appreciated. thank you


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.