# Regarding the objective of PPO training in instructGPT

The following objective is taken from the paper 'Training language models to follow instructions with human feedback':which is used to fine-tune the pre-trained language model using Proximal Policy Optimization (PPO). In the original paper, the objective of PPO is as follows: comparing the two objectives we can see the term with beta in equation 2 must be the KL term in equation 5. My question is, why is the KL term in equation 2 computed with respect to (x,y)? Shouldn't it be with respect to $$\phi$$ which parameterizes the policy $$\pi$$?

• x/y are input/output, not the nn parameters... Commented Feb 28, 2023 at 23:05
• i mean that is my point. Policy should be parameterized by $\pi$, not (x,y)
– Sam
Commented Mar 4, 2023 at 2:09
• no, $\pi$ is the policy, which is parametrized by some parameters, they have just changed the notation from $\theta$ to $\phi$ Commented Mar 4, 2023 at 11:25
• The instructgpt paper forgot the first ratio, as in the PPO paper, haven’t they? Commented Mar 13 at 16:06

• I don't quite follow $E_{(x, y) \sim D_{\pi_\phi^{\mathrm{RL}}}}\left[-\beta \log \left(\pi_\phi^{\mathrm{RL}}(y \mid x) / \pi^{\mathrm{SFT}}(y \mid x)\right)\right]$ = $-\beta E_{(x, y) \sim D_{\pi_\phi^{\mathrm{RL}}}}\left[\log \left(\pi_\phi^{\mathrm{RL}}(y \mid x) / \pi^{\mathrm{SFT}}(y \mid x)\right)\right]$ = $-\beta \mathrm{KL}(\pi_\phi^{\mathrm{RL}} | \pi^{\mathrm{SFT}})$ While the term in the PPO equation is effectively $-\beta \mathrm{KL}(\pi^{\mathrm{SFT}} |\pi_\phi^{\mathrm{RL}} )$ ? Commented Mar 28, 2023 at 13:11