The following objective is taken from the paper 'Training language models to follow instructions with human feedback':enter image description herewhich is used to fine-tune the pre-trained language model using Proximal Policy Optimization (PPO). In the original paper, the objective of PPO is as follows: enter image description herecomparing the two objectives we can see the term with beta in equation 2 must be the KL term in equation 5. My question is, why is the KL term in equation 2 computed with respect to (x,y)? Shouldn't it be with respect to $\phi$ which parameterizes the policy $\pi$?

  • $\begingroup$ x/y are input/output, not the nn parameters... $\endgroup$
    – Alberto
    Commented Feb 28, 2023 at 23:05
  • $\begingroup$ i mean that is my point. Policy should be parameterized by $\pi$, not (x,y) $\endgroup$
    – Sam
    Commented Mar 4, 2023 at 2:09
  • $\begingroup$ no, $\pi$ is the policy, which is parametrized by some parameters, they have just changed the notation from $\theta$ to $\phi$ $\endgroup$
    – Alberto
    Commented Mar 4, 2023 at 11:25
  • $\begingroup$ The instructgpt paper forgot the first ratio, as in the PPO paper, haven’t they? $\endgroup$
    – Nathan G
    Commented Mar 13 at 16:06

1 Answer 1


I think I figured it out. The KL is with respect to the distribution of the actions given a stochastic policy, so it makes sense that int the instuctGPT paper the expectation is w.r.t. (x,y). In the PPO paper the expectation w.r.t. distribution of actions is implicit in the 'KL' function, and the expectation outside is to average across an entire episode.Since in the formulation of instructGPT there is only one step per episode (more akin to bandit setting), this is not needed.

  • 1
    $\begingroup$ I don't quite follow $E_{(x, y) \sim D_{\pi_\phi^{\mathrm{RL}}}}\left[-\beta \log \left(\pi_\phi^{\mathrm{RL}}(y \mid x) / \pi^{\mathrm{SFT}}(y \mid x)\right)\right]$ = $-\beta E_{(x, y) \sim D_{\pi_\phi^{\mathrm{RL}}}}\left[\log \left(\pi_\phi^{\mathrm{RL}}(y \mid x) / \pi^{\mathrm{SFT}}(y \mid x)\right)\right]$ = $-\beta \mathrm{KL}(\pi_\phi^{\mathrm{RL}} | \pi^{\mathrm{SFT}})$ While the term in the PPO equation is effectively $-\beta \mathrm{KL}(\pi^{\mathrm{SFT}} |\pi_\phi^{\mathrm{RL}} )$ ? $\endgroup$
    – Tinatim
    Commented Mar 28, 2023 at 13:11
  • $\begingroup$ I think SFT is the old policy in PPO. $\endgroup$
    – Sam
    Commented Mar 29, 2023 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.