The following objective is taken from the paper 'Training language models to follow instructions with human feedback':which is used to fine-tune the pre-trained language model using Proximal Policy Optimization (PPO). In the original paper, the objective of PPO is as follows: comparing the two objectives we can see the term with beta in equation 2 must be the KL term in equation 5. My question is, why is the KL term in equation 2 computed with respect to (x,y)? Shouldn't it be with respect to $\phi$ which parameterizes the policy $\pi$?
I think I figured it out. The KL is with respect to the distribution of the actions given a stochastic policy, so it makes sense that int the instuctGPT paper the expectation is w.r.t. (x,y). In the PPO paper the expectation w.r.t. distribution of actions is implicit in the 'KL' function, and the expectation outside is to average across an entire episode.Since in the formulation of instructGPT there is only one step per episode (more akin to bandit setting), this is not needed.