REINFORCE: how to sample the derivative When implementing the REINFORCE algorithm, it requires the derivative of the log probability. As highlighted in the image below (from David Silver's 7th lecture).

In many implementations I've seen people sample the action, find the probability and then take the log. My question is how to interpret this operator ∇ in implementation.
How do we sample this score function?
Thanks
 A: Policy gradient methods (like REINFORCE) optimise a policy function (which I've called $P$). [Note a function in Maths is very similar to a function (aka a method) in most programming languages.] The inputs to this function are a state and a vector of parameters. The outputs are the probability of taking each of the legal actions in that state.
The first thing to note is that this is a method for episodic tasks i.e. tasks that always finish in a finite number of steps. The next thing to note is that this is an on-policy method; the experience used in the learning is generated by current version of the policy you are trying to improve. This also means you must have the policy function (i.e. the actual code that generates the actions). You are also going to need a partial derivative function. The inputs to this are an action and the current parameter values; the outputs are the partial derivatives of log probability of that action with respect to the parameters evaluated at the current values of the parameters. In the "good old days" you had to get a formula for the derivative with pen and paper, and convert it to code. Usually the form of the policy function was chosen so these pen and paper manipulations were relatively simple. But now automatic symbolic differentiation (e.g in Tensorflow) is an option as well.
To generate an episode, repeatedly

*

*Sample the next state from state transition probabilities i.e. simulate the environment.

*Calculate the policy function in the current state. Then draw an action from the probability distribution returned.

until the final state is reached and the episode is over.
Then to learn from that episode, for each action

*

*Calculate the derivative function for the action chosen and current parameters.

*Perform the update i.e. the equation you highlighted.
Conveniently, each action can be considered seperately.

I've rewritten the highlighted equation in slightly different notation with an explanation of each term, as that might help.
$\theta_{i,e+1} = \theta_{i,e}+\alpha\left[ \frac{\partial }{\partial \theta_i}log\left(P\left(a_{e,t}|s_{e,t},\theta_e\right)\right)\right] v_{e,t}$
where
$P$ is the policy function. The inputs are the state  $s_{e,t}$ (i.e. the information available at the time of the decision) at decision $t$ in episode $e$ and the values of the parameters (aka weights) $\theta_e$ at the start of the episode $e$. The output is the probability of taking any of the legal actions in that state.
$P\left(a_{e,t}|s_{e,t},\theta_e\right)$ is the probability of the action actually taken at decision $t$ in episode $e$.
$\frac{\partial }{\partial \theta_i}$ is the derivative with respect to the $i^{th}$ parameter. $\left[ \frac{\partial }{\partial \theta_i}log\left(P\left(a_{e,t}|s_{e,t},\theta_e\right)\right)\right]$ is calculated at the current value of the parameters $\theta$.
$v_{e,t}$ is an unbiased estimate of the expected value after taking $t^{th}$ action in episode $e$. The simplest version is to use the overall reward for the episode $R_e$.
Hopefully, I have explained whichever bit of notation was causing your confusion. It might also be useful to do a search for REINFORCE [and Williams as author] to get notes by other lecturers on these baseline reinforcement comparison methods.
