In a sparse reward problem, is it possible to remove reward shaping once the RL agent trains long enough to consistently reach the final reward?

Question

I'm new to machine learning, and primarily looking at it from the perspective of it's applications to control theory.

In the application found in this paper, a RL agent attempts to land a spacecraft near a target. The optimization environment is very sparse, with a large reward gained if the spacecraft can successfully land, along with small penalties for fuel use along the way. In order to overcome the sparsity and allow the RL agent to successfully land, the authors employ reward shaping and eventually achieve a successful agent.

From this point, the authors discuss improving the reward shaping function to increase fuel efficiency. However, couldn't they just continue training this same agent without any shaping, thus allowing the fuel efficiency rewards alone to shape training?

I'm very new to the field, so I suspect I'm missing some fundamental or practical issue with this approach.

Answer 1

It seems that the hyperlink is broken, so for now, I will attempt to clarify some details without having read the paper and can answer the specifics later.

---

I suspect that even though the agent has found a solution, it may not be the optimal solution and that finding the optimal solution may still require a whole bunch of extra exploration in both state and action space. Reward-shaping is one way of exploring efficiently, and useful priors such as fuel efficiency may allow the agent to find the optimal solution quicker even after having found the reward. That being said, I think that in the case of 2D navigation in a grid-like world, what you are proposing may work totally fine as long as the discount factor $\gamma \neq 1$ which would effectively be the same as fuel efficiency in the spacecraft game.

---

Since you said that you're new to the field, here's a more detailed answer to make sure that we're on the same page re. your question.

When the rewards are sparse and the state-space is very large, RL agents are often unable to learn because there is a very low probability of reaching the rewarding state(s) by random exploration in a finite amount of time. This problem is best known in the game Montezuma's revenge, where Deep RL agents without tweaks (vanilla DQN; Mnih et al 2015) have struggled to perform well (though recent attempts have fared well). Ideally, an agent should learn to explore the state-space better, see whether you have visited some states before, and if not, go visit those to reduce your uncertainty about them and see whether they are rewarding. People use prioritized experience replays, reward shaping, uncertainty-based exploration, and a bunch of other tweaks to solve this. The point of these tweaks is to help the agent with the exploration problem such that it can.

However, once the agent learns to find the correct area of the state-space, i.e. one that leads it to the rewarding locations which are sparse, you could remove the bonuses that you artificially constructed, for instance, by reward-shaping since the exploration problem is not as difficult because the agent knows where the reward is.

If I understand correctly, this is the logic you are using when you framed your question.