# Any example code of REINFORCE algorithm proposed by Williams?

Does any one know example of an Algorithm Williams proposed in Paper "A class of gradient-estimating algorithms for reinforcement learning in neural networks" http://incompleteideas.net/sutton/williams-92.pdf

• I will forget to come here when I do, But I am just about to finish refactoring openAI's github.com/pytorch/examples/blob/master/reinforcement_learning/… to an object oriented style, which is actually scalable. ping me on that and I will post it here Jan 15, 2019 at 23:07
• I have it implemented here: Lua code. Hope it helps! Mar 25, 2019 at 23:28

From David Silver's RL lecture on Policy Gradient methods, slide 21 here is pseudo-code for the episodic Reinforce algorithm, which basically is a gradient-based method where the expected return is sampled directly from the episode (as opposed to estimating it with some learned function). In this case the expected return is actually the total episodic reward onward that step, $G_t$.

initialise $\theta$

for each episode {$s_1, a_1, r_2 ... s_{T-1}, a_{T-1}, r_T$} sampled from policy $\pi_\theta$ do

for t = 1 to T - 1 do

$\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi_\theta(s_t,a_t) G_t$

end for

end for

This algorithm suffers from high variance because the sampled rewards can be very different from one episode to another therefore this algorithm is usually used with a baseline substracted from the policy. Here is a more detailed explanation complete with code samples.

• I am curious why update weights each timestep instead of update once in the end of episode? My understanding is $\theta$ is not changed in forward pass of the whole trajectrory Mar 25, 2018 at 13:05
• @eric2323223 David Silver's course (recommended) discusses that throughly. REINFORCE is a monte-carlo method. You could do a more frequent update, which is better for many cases. Go watch the videos to get much better explanations than what I can give here. In general, he calls the more frequent method "Temporal difference" or, "TD", with an optimization called "TD(lambda)" Jan 15, 2019 at 23:10
• Is REINFORCE a on-policy algorithm? Dec 15, 2019 at 1:51

The REINFORCE algorithm for policy-gradient reinforcement learning is a simple stochastic gradient algorithm. It works well when episodes are reasonably short so lots of episodes can be simulated. Value-function methods are better for longer episodes because they can start learning before the end of a single episode.