Reinforcement learning by Sutton, Tic tac toe self play

Question

I just started Sutton and Barto's book, Reinforcement Learning: An Introduction, and am curious as to how to think about the answer to Exercise 1.1: Self-Play. Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself. What do you think would happen in this case? Would it learn a different way of playing?

One could also think of the following related sub-questions, but they haven't made my thoughts any clearer.

1. Would removing the random part of the learning change the situation- i.e. always following optimal policy and not exploring?
2. How would it depend on who is the first mover?

Answer 1

I am not sure about the first question. Regarding the second, these are my thoughts:

If you think about the state space of tic-tac-toe, it can be partitioned into two mutually exclusive subsets, one consisting of states seen by the agent when playing first, the other consisting of states seen while playing second. If one of the sides is always going to play first, then the other side will experience only one of the two subsets in the state-space. It would try to learn a policy that would try to win as a second player.

It would be good to have both the sides play as first and second players. Toss a coin before every match - if heads, let the left-side play first, else the right-side starts. This way we can at least ensure that the agent's policy is independent of which side starts first.