I would like to ask is it possible to train an agent under self-playing setting but with just one policy to be trained? What are the foreseeable problems with such an implementation? My concern is as such: suppose agent A starts off as in role 1, and later goes role 2 (which is role 1's opponent) after a few iteration. When A switches to role 2, it might use the information gained when A was in role 1 to its current role's advantage. And after another few iterations when A switches back to role 1, A might use the information gained when A was in role 2 to its current role's advantage. This is because we assume there is just one policy for agent A, so the weights are updated and kept when A switches its role. The information gained can be things such as new strategies learnt, new information gained (when the agent's knowledge of the environment is incomplete), etc. So a more sensible way of training I think would be using two agents A and B with two separate policies to train - more like in a generative adversarial network setting in some sense. But then by definition this is not considered as self-play isn't it?

Add-on: So relating to the case of Alphazero playing chess, if the agent is really just playing with itself, despite chess belongs to the so-called perfect information game, but the opponent's thoughts/strategy/decision making process should be still unknown to the player. But if my assumption of how self-playing is true, then the agent A will have the thoughts/strategy/decision making process of role 2 when it is in role 1. Isn't that cheating (because in reality those knowledge are not attainable; rather one can only guess what are those information from its opponents)? (I haven't read through the Alphazero paper yet because I am not proficient enough to understand those technical details, so I would really appreciate it if anyone could explain the relevant part to me)


1 Answer 1


If you switch policy under $A$ and $B$ or have a single policy where the first parameter for lookup is either $A$ or $B$, then it is the same.

It will only matter in terms of coding where you want to generalize across $A$ and $B$ roles and how you want the models laid out.

  • $\begingroup$ So does Alphazero use this way do to training as well? $\endgroup$
    – Sam
    Feb 12, 2020 at 2:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.