I'm trying to make sense of the "The Bandit Gradient Algorithm as Stochastic Gradient Ascent" proof in Sutton and Barto's intro to RL textbook. I'm stuck on the line

$E[(q_*(A_t)-B_t)\frac{\partial\pi_t(A_t)}{\partial H_t(a)}/\pi_t(A_t)] = E[(R_t-\bar{R}_t)\frac{\partial\pi_t(A_t)}{\partial H_t(a)}/\pi_t(A_t)]$$

They say they can replace $q_*(A_t)$ with $R_t$ because $E[R_t|A_t]=q_*(A_t)$, but I can't figure out how to show this rigorously.


1 Answer 1


It's the definition of the optimal $Q$-function: $$ Q* = \mathbb{E}[R_t|S_t,A_t] $$

(the state is omitted as in Bandits there is only 1 state)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.