First, it's often cost prohibitive for humans to gather enough labels to meaningfully train such large deep learning models. The reward model therefore replaces a human for a high percentage of evaluations to save time and money. Second, continuously adding new labels, based on feedback from humans throughout training, also prevents the model from fixating on strange, unforeseeable behaviors not covered by the original set of labels.
A core concept of Instruct GPT (and therefore Chat GPT and similarly inspired LLMs) is Reinforcement Learning with Human Feedback (RLHF). DeepMind and OpenAI collaborated on the RLHF research, and published the paper Deep Reinforcement Learning from Human Preferences.
In their paper, the authors state the drawback of your idea:
An alternative approach is to allow a human to provide feedback on our system’s current behavior and to use this feedback to define the task. In principle this fits within the paradigm of reinforcement learning, but using human feedback directly as a reward function is prohibitively expensive for RL systems that require hundreds or thousands of hours of experience. In order to practically train deep RL systems with human feedback, we need to decrease the amount of feedback required by several
orders of magnitude.
DeepMind adds further details on their RLHF landing page:
The design also does not put an onerous burden on the human operator, who only has to review around 0.1% of the agent’s behaviour to get it to do what they want. However, this can mean reviewing several hundred to several thousand pairs of clips, something that will need to be reduced to make it applicable to real world problems.
So even with the reward model taking 99.9% of evaluations there is still an impractical amount of human effort required.
Even if cost were not an issue, the authors of the RLHF paper point out that the policy model (the LLM in the context of Instruct GPT) tends to overfit to the original human labels if you don't provide new human labels throughout training. Supervised learning differs from RL in that supervised learning expects all the labels upfront whereas RL provides new labels based on how the model interacts with its environment, in this case the user.
The authors tried a variety of changes to their training process. One variation they described as follows:
We train on queries only gathered at the beginning of training, rather than gathered throughout training (no online queries).
The authors later note that:
Of particular interest is the poor performance of offline reward predictor training; here we find
that due to the nonstationarity of the occupancy distribution, the predictor captures only part of the
true reward, and maximizing this partial reward can lead to bizarre behavior that is undesirable as
measured by the true reward
An illustrative example from the paper involves the game of Pong. The model learns to not lose but not win either, playing an infinite volley. You can see the results in the paper's graph, shown below. Using only labels from the beginning, like in supervised learning, generally produces the worst results: