I just started Sutton and Barto's book, Reinforcement Learning: An Introduction, and am curious as to how to think about the answer to Exercise 1.1: Self-Play. Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself. What do you think would happen in this case? Would it learn a different way of playing?

One could also think of the following related sub-questions, but they haven't made my thoughts any clearer.

  1. Would removing the random part of the learning change the situation- i.e. always following optimal policy and not exploring?
  2. How would it depend on who is the first mover?

I am not sure about the first question. Regarding the second, these are my thoughts:

If you think about the state space of tic-tac-toe, it can be partitioned into two mutually exclusive subsets, one consisting of states seen by the agent when playing first, the other consisting of states seen while playing second. If one of the sides is always going to play first, then the other side will experience only one of the two subsets in the state-space. It would try to learn a policy that would try to win as a second player.

It would be good to have both the sides play as first and second players. Toss a coin before every match - if heads, let the left-side play first, else the right-side starts. This way we can at least ensure that the agent's policy is independent of which side starts first.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.