Why do language models like InstructGPT and LLM utilize reinforcement learning instead of supervised learning to learn based on user-ranked examples? Why do language models like InstructGPT and LLM utilize reinforcement learning instead of supervised learning to learn based on user-ranked examples?
Language models like InstructGPT and ChatGPT are initially pretrained using self-supervised methods, followed by supervised fine-tuning. The researchers then train a reward model on responses that are ranked by humans on a scale of 1 to 5. After the reward model has been trained using supervised learning, reinforcement learning with proximal policy optimization (PPO) is used to update the language model.
Why do they then use RLL with PPO to update the LLM based on the reward signal. Why don't they train / fine-tune the LLM using the labels (ranks 1-5 provided by humans) in a supervised fashion using ranking or (ordinal) regression losses?
 A: Your LLM will give you a categorical distribution as output, over which you can sample, and thus use RL to estimate the gradient...
What you are suggesting looks more like a GAN which instead of using a discriminator, you have a ordinal-NN, over which you take the gradient to maximize the ordinal output... however, this is unstable and usually the generator (LLM) will have a very easy time maximizing the output of the discriminator, which is known as mode alignment...
You can probably fine tune a LLM with a loss like:
$$
\nabla L(\theta) = \nabla(-D_{kl}(\pi_{\theta}(y|x)||\pi_{orig}(y|x))) + \nabla D(\pi_{\theta}(y|x)|x)
$$
where the second term is the gradient flowing from the conditional discriminator (which maximization should give you more human-like response), and the first one is just a penalization term to not go too far from the pre-trained model
However, in my opinion, this will just make the LLM overfit the discriminator...
A: Supervised LLM training only gives the model positive examples, i.e. ones it should produce. It does not provide the negative ones, and a naive attempt to do so would probably fail due to the sheer volume of negatives in the space of possible outputs.
Indeed, you probably could somehow penalize the model for producing outputs like "afsjkafnkfkasfjk nasjfasfas" but that would be a poor negative sample as the model would probably not produce this gibberish in the first place. Coming up with a particular set of useful negative examples is hard and probably depends on a particular model. This is where RL comes in: it allows you to operate on the models' outputs themselves, which is exactly the thing you want to improve.
