Why do language models like InstructGPT and LLM utilize reinforcement learning instead of supervised learning to learn based on user-ranked examples?

Question

Why do language models like InstructGPT and LLM utilize reinforcement learning instead of supervised learning to learn based on user-ranked examples?

Language models like InstructGPT and ChatGPT are initially pretrained using self-supervised methods, followed by supervised fine-tuning. The researchers then train a reward model on responses that are ranked by humans on a scale of 1 to 5. After the reward model has been trained using supervised learning, reinforcement learning with proximal policy optimization (PPO) is used to update the language model.

Why do they then use RLL with PPO to update the LLM based on the reward signal. Why don't they train / fine-tune the LLM using the labels (ranks 1-5 provided by humans) in a supervised fashion using ranking or (ordinal) regression losses?

Answer 1

Supervised LLM training only gives the model positive examples, i.e. ones it should produce. It does not provide the negative ones, and a naive attempt to do so would probably fail due to the sheer volume of negatives in the space of possible outputs.

Indeed, you probably could somehow penalize the model for producing outputs like "afsjkafnkfkasfjk nasjfasfas" but that would be a poor negative sample as the model would probably not produce this gibberish in the first place. Coming up with a particular set of useful negative examples is hard and probably depends on a particular model. This is where RL comes in: it allows you to operate on the models' outputs themselves, which is exactly the thing you want to improve.

Answer 2

The paper LIMA: Less Is More for Alignment uploaded a few days ago to arXiv shows that fine-tuning with the standard supervised loss without any reinforcement learning works fine:

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.

Also note that the traditional reinforcement learning from human feedback (RLHF) process can be simplified, according to the DPO paper: Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. arXiv:2305.18290:

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper, we leverage a mapping between reward functions and optimal policies to show that this constrained reward maximization problem can be optimized exactly with a single stage of policy training, essentially solving a classification problem on the human preference data. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant and computationally lightweight, eliminating the need for fitting a reward model, sampling from the LM during fine-tuning, or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds RLHF's ability to control sentiment of generations and improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

Here is a nice blogpost by HuggingFace on fine-tuning Llama 2 with DPO.

Answer 3

Your LLM will give you a categorical distribution as output, over which you can sample, and thus use RL to estimate the gradient...
What you are suggesting looks more like a GAN which instead of using a discriminator, you have a ordinal-NN, over which you take the gradient to maximize the ordinal output... however, this is unstable and usually the generator (LLM) will have a very easy time maximizing the output of the discriminator, which is known as mode alignment...
You can probably fine tune a LLM with a loss like: $$ \nabla L(\theta) = \nabla(-D_{kl}(\pi_{\theta}(y|x)||\pi_{orig}(y|x))) + \nabla D(\pi_{\theta}(y|x)|x) $$ where the second term is the gradient flowing from the conditional discriminator (which maximization should give you more human-like response), and the first one is just a penalization term to not go too far from the pre-trained model

However, in my opinion, this will just make the LLM overfit the discriminator...

Answer 4

Short Answer:

First, it's often cost prohibitive for humans to gather enough labels to meaningfully train such large deep learning models. The reward model therefore replaces a human for a high percentage of evaluations to save time and money. Second, continuously adding new labels, based on feedback from humans throughout training, also prevents the model from fixating on strange, unforeseeable behaviors not covered by the original set of labels.

Long Answer:

A core concept of Instruct GPT (and therefore Chat GPT and similarly inspired LLMs) is Reinforcement Learning with Human Feedback (RLHF). DeepMind and OpenAI collaborated on the RLHF research, and published the paper Deep Reinforcement Learning from Human Preferences.

In their paper, the authors state the drawback of your idea:

An alternative approach is to allow a human to provide feedback on our system’s current behavior and to use this feedback to define the task. In principle this fits within the paradigm of reinforcement learning, but using human feedback directly as a reward function is prohibitively expensive for RL systems that require hundreds or thousands of hours of experience. In order to practically train deep RL systems with human feedback, we need to decrease the amount of feedback required by several orders of magnitude.

DeepMind adds further details on their RLHF landing page:

The design also does not put an onerous burden on the human operator, who only has to review around 0.1% of the agent’s behaviour to get it to do what they want. However, this can mean reviewing several hundred to several thousand pairs of clips, something that will need to be reduced to make it applicable to real world problems.

So even with the reward model taking 99.9% of evaluations there is still an impractical amount of human effort required.

Even if cost were not an issue, the authors of the RLHF paper point out that the policy model (the LLM in the context of Instruct GPT) tends to overfit to the original human labels if you don't provide new human labels throughout training. Supervised learning differs from RL in that supervised learning expects all the labels upfront whereas RL provides new labels based on how the model interacts with its environment, in this case the user.

The authors tried a variety of changes to their training process. One variation they described as follows:

We train on queries only gathered at the beginning of training, rather than gathered throughout training (no online queries).

The authors later note that:

Of particular interest is the poor performance of offline reward predictor training; here we find that due to the nonstationarity of the occupancy distribution, the predictor captures only part of the true reward, and maximizing this partial reward can lead to bizarre behavior that is undesirable as measured by the true reward

An illustrative example from the paper involves the game of Pong. The model learns to not lose but not win either, playing an infinite volley. You can see the results in the paper's graph, shown below. Using only labels from the beginning, like in supervised learning, generally produces the worst results: