In the InstructGPT paper they define objective of RL stage as:
They try to maximize this objective using PPO.
I have trouble understanding how they plug this objective into the PPO though. Do they just replace each $r_t$ in the computation of PPO advantages $\hat{A}_t$ by the value of this objective?
(which would also mean that they use $L^\text{CLIP}$ instead of $L^\text{KLPEN}$ in the final loss, otherwise KL term would be kind of duplicated, won't it?)
InstructGPT's detailed algorithm implementation is not open sourced you may continue to search any other paper talking about its detailed PPO styled implementation. Having said that, your reference does mention that:
Finally, since the RM loss is invariant to shifts in reward, we normalize the reward model using a bias so that the labeler demonstrations achieve a mean score of 0 before doing RL.
Once again following Stiennon et al. (2020), we fine-tuned the SFT model on our environment using PPO (Schulman et al., 2017). The environment is a bandit environment which presents a random customer prompt and expects a response to the prompt. Given the prompt and response, it produces a reward determined by the reward model and ends the episode... For "PPO" models, γ is set to 0. Unless otherwise specified, in this paper InstructGPT refers to the PPO-ptx models.
Since the InstructGPT PPO agent is facing a (contextual) bandit environment and it's already trained based on the supervised fine-tuning (SFT) large language GPT-3 model with the default starting policy $\pi^{SFT}$, it might be much simpler to be implemented as a PPO algo whose RL objective function in this bandit case is just your referenced equation (2) without any state transition and timesteps $T$ ($T=1$ in this case). And also because reward is normalized before training as claimed above, the advantage function $\hat A$ of any bandit arm could be simply replaced by the term $r_\theta(x,y)-\beta \log(\pi_\Phi^{RL}(y|x)/\pi^{SFT}(y|x))$ defined in your equation (2) withuot any state value baseline. Then you can run PPO with its surrogate objective function either as CLIP or KLPEN or something else.
