0
$\begingroup$

I am trying to understand a step in the proof of the "Bandit Gradient Algorithm As Stochastic Gradient Ascent" in Suttons Book (http://incompleteideas.net/book/RLbook2018.pdf) at page 39. Starting from:

$\frac{\partial E[R_{t}]}{\partial H_{t}(a)} = E \left[ (q_{*}(A_{t}) - B_{t}) \frac{\partial \pi_{t}(A_{t})}{\partial H_{t}(a)} \frac{1}{\pi_{t}(A_{t})} \right]$

where $q_{*}(A_{t}) = E[R_{t} \vert A_{t}]$ and $\pi_{t}(A_{t}) = \frac{e^{H_{t}(a)}}{\sum\limits_{b=1}^{k} e^{H_{t}(b)}}$

we get the following line:

$\frac{\partial E[R_{t}]}{\partial H_{t}(a)} = E \left[ (R_{t} - B_{t}) \frac{\partial \pi_{t}(A_{t})}{\partial H_{t}(a)} \frac{1}{\pi_{t}(A_{t})} \right]$

I am trying to derive this line by using the "law of iterated expections" ($E[E[X\vert Y]] = E[X]$) but without success. I am wondering how this was working.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.