# Any example code of REINFORCE algorithm proposed by Williams?

Does any one know example of an Algorithm Williams proposed in Paper "A class of gradient-estimating algorithms for reinforcement learning in neural networks" http://incompleteideas.net/sutton/williams-92.pdf

From David Silver's RL lecture on Policy Gradient methods, slide 21 here is pseudo-code for the episodic Reinforce algorithm, which basically is a gradient-based method where the expected return is sampled directly from the episode (as opposed to estimating it with some learned function). In this case the expected return is actually the total episodic reward onward that step, $G_t$.

initialise $\theta$

for each episode {$s_1, a_1, r_2 ... s_{T-1}, a_{T-1}, r_T$} sampled from policy $\pi_\theta$ do

for t = 1 to T - 1 do

$\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi_\theta(s_t,a_t) G_t$

end for

end for

This algorithm suffers from high variance because the sampled rewards can be very different from one episode to another therefore this algorithm is usually used with a baseline substracted from the policy. Here is a more detailed explanation complete with code samples.

• I am curious why update weights each timestep instead of update once in the end of episode? My understanding is $\theta$ is not changed in forward pass of the whole trajectrory – eric2323223 Mar 25 '18 at 13:05
• @eric2323223 David Silver's course (recommended) discusses that throughly. REINFORCE is a monte-carlo method. You could do a more frequent update, which is better for many cases. Go watch the videos to get much better explanations than what I can give here. In general, he calls the more frequent method "Temporal difference" or, "TD", with an optimization called "TD(lambda)" – Gulzar Jan 15 '19 at 23:10
• Is REINFORCE a on-policy algorithm? – GoingMyWay Dec 15 '19 at 1:51

The REINFORCE algorithm for policy-gradient reinforcement learning is a simple stochastic gradient algorithm. It works well when episodes are reasonably short so lots of episodes can be simulated. Value-function methods are better for longer episodes because they can start learning before the end of a single episode.