# Multi-agent actor-critic MADDPG algorithm confusion

I am trying to understand the paper from openAI called Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

In the paper, they mention that they combat the problem of environment non-stationarity by sampling from sub-policies:

I am confused about: (1) how subsampling would resolve the non-stationarity problem and (2) why would the individual agents have more than one possible (sub) policy - shouldn't there be a single optimum policy for each agent?

(1) how subsampling would resolve the non-stationarity problem

The idea about sampling a variety of sub-policies for other agents to execute during training is that this introduces more variety in the behaviour of competing agents, rather than always only training against the single most recent "version" of opponents (which can result in "overfitting" against those agents). If there is variety in the behaviour of opponents, your agent will be forced to try learning a robust policy in the sense that it will try learning a policy that can handle all opponents. Without that variety, if you would only always select the most recent versions of opponents, your agent would instead be incentivized to only learn a policy that is strong against those most recent versions of opponents.

Consider, for example, the game of Rock-Paper-Scissors. Let $$P_1$$ and $$P_2$$ denote two agents that are simultaneously learning. Suppose that they would only ever train against each other (rather than having more varied training partners through sampling). Suppose $$P_1$$ is randomly initialized to mostly just play Rock, and $$P_2$$ is randomly initialized to mostly just play Paper.

$$P_2$$ will initially win most of its games, and $$P_1$$ will then learn to just play Scissors very often. Once $$P_1$$ has learned that, $$P_2$$ will start learning to play Rock very often. Once that is done, $$P_1$$ will start learning to play Paper very often. Both agents will just keep going in circles like that, always learning only to counter the most recent behaviour of the other player.

If we instead introduce more variety in training partners by sampling from an ensemble of multiple learned policies, we will be more likely to converge to the optimal strategy of selecting actions uniformly at random; that's the only strategy that will be likely to perform well against an ensemble of varyious policies.

(2) why would the individual agents have more than one possible (sub) policy - shouldn't there be a single optimum policy for each agent?

Ultimately we'll often want to converge to a single*, optimal policy for every agent, yes. But normally we don't have that yet... that's why we're doing Reinforcement Learning in the first place! We don't know what an optimal (or even just a good) policy looks like, we have to learn that first. During that learning process, if we want to (which we do based on the reasoning in my answer to your previous question above), we can easily just learn an ensemble of different policies, rather than learning a single policy. This can, for example, be done simply by training each sub-policy on a different subset of the experience that we collect.