I would like to ask is it possible to train an agent under self-playing setting but with just one policy to be trained? What are the foreseeable problems with such an implementation? My concern is as such: suppose agent A starts off as in role 1, and later goes role 2 (which is role 1's opponent) after a few iteration. When A switches to role 2, it might use the information gained when A was in role 1 to its current role's advantage. And after another few iterations when A switches back to role 1, A might use the information gained when A was in role 2 to its current role's advantage. This is because we assume there is just one policy for agent A, so the weights are updated and kept when A switches its role. The information gained can be things such as new strategies learnt, new information gained (when the agent's knowledge of the environment is incomplete), etc. So a more sensible way of training I think would be using two agents A and B with two separate policies to train - more like in a generative adversarial network setting in some sense. But then by definition this is not considered as self-play isn't it?
Add-on: So relating to the case of Alphazero playing chess, if the agent is really just playing with itself, despite chess belongs to the so-called perfect information game, but the opponent's thoughts/strategy/decision making process should be still unknown to the player. But if my assumption of how self-playing is true, then the agent A will have the thoughts/strategy/decision making process of role 2 when it is in role 1. Isn't that cheating (because in reality those knowledge are not attainable; rather one can only guess what are those information from its opponents)? (I haven't read through the Alphazero paper yet because I am not proficient enough to understand those technical details, so I would really appreciate it if anyone could explain the relevant part to me)
