The following objective is taken from the paper 'Training language models to follow instructions with human feedback':enter image description herewhich is used to fine-tune the pre-trained language model using Proximal Policy Optimization (PPO). In the original paper, the objective of PPO is as follows: enter image description herecomparing the two objectives we can see the term with beta in equation 2 must be the KL term in equation 5. My question is, why is the KL term in equation 2 computed with respect to (x,y)? Shouldn't it be with respect to $\phi$ which parameterizes the policy $\pi$?

  • $\begingroup$ x/y are input/output, not the nn parameters... $\endgroup$
    – Alberto
    Feb 28, 2023 at 23:05
  • $\begingroup$ i mean that is my point. Policy should be parameterized by $\pi$, not (x,y) $\endgroup$
    – Sam
    Mar 4, 2023 at 2:09
  • $\begingroup$ no, $\pi$ is the policy, which is parametrized by some parameters, they have just changed the notation from $\theta$ to $\phi$ $\endgroup$
    – Alberto
    Mar 4, 2023 at 11:25

1 Answer 1


I think I figured it out. The KL is with respect to the distribution of the actions given a stochastic policy, so it makes sense that int the instuctGPT paper the expectation is w.r.t. (x,y). In the PPO paper the expectation w.r.t. distribution of actions is implicit in the 'KL' function, and the expectation outside is to average across an entire episode.Since in the formulation of instructGPT there is only one step per episode (more akin to bandit setting), this is not needed.

  • 1
    $\begingroup$ I don't quite follow $E_{(x, y) \sim D_{\pi_\phi^{\mathrm{RL}}}}\left[-\beta \log \left(\pi_\phi^{\mathrm{RL}}(y \mid x) / \pi^{\mathrm{SFT}}(y \mid x)\right)\right]$ = $-\beta E_{(x, y) \sim D_{\pi_\phi^{\mathrm{RL}}}}\left[\log \left(\pi_\phi^{\mathrm{RL}}(y \mid x) / \pi^{\mathrm{SFT}}(y \mid x)\right)\right]$ = $-\beta \mathrm{KL}(\pi_\phi^{\mathrm{RL}} | \pi^{\mathrm{SFT}})$ While the term in the PPO equation is effectively $-\beta \mathrm{KL}(\pi^{\mathrm{SFT}} |\pi_\phi^{\mathrm{RL}} )$ ? $\endgroup$
    – Tinatim
    Mar 28, 2023 at 13:11
  • $\begingroup$ I think SFT is the old policy in PPO. $\endgroup$
    – Sam
    Mar 29, 2023 at 3:18

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .