# How to stabilize training in multi-objective reinforcement learning?

I am trying to train an agent to maximize multi-objectives. If I just add up rewards from different objectives, my problem is that the agent maximizes the 'easy' objective, at the expense of the hard one. How can I adaptively penalize the easiness with which the agent realize an objective?

## 2 Answers

If the reward from each objective $1... k$ is $r_1, r_2, ... r_k$ set the goal to be maximization of $R = \min_i r_i$ instead of $R = \sum_i r_i$. By maximizing the minimum reward across all goals, the agent will be forced to learn all objectives in order to perform better.

A possibly better behaved alternative would be to set $R = -|\text{softmax}(-\vec r)|_{L_1}$ (this sets the reward to be the negative softmax of the cost). This should accomplish about the same thing while being a bit smoother.

The simplest approach would be to maximise a weighted sum of the objectives. By giving a larger weight to the harder-to-achieve objectives you encourage the agent to place more emphasis on them. Finding the correct weights to give the desired trade-off between can be difficult though, and sometimes may not be possible as there may be no weights which actually result in certain solutions being obtained.

The alternative is to use a non-linear scalarisation of the objectives as suggested in shimao's answer. However that may cause convergence problems for reinforcement learning methods like Q-learning which are based on the Bellman equation.

I co-authored a survey paper a few years ago which provides a detailed discussion of these issues, which you may find useful: https://arxiv.org/abs/1402.0590

• This paper lacks practical examples of scalarisations, although the theory is interesting – Mostafa Aug 21 '17 at 8:34