This is regarding the first exercise in Sutton and Barto's book on Reinforcement Learning. I have a few questions about the exercise; I am not looking for an exact solution.
Exercise 1.1: Self-Play Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself, with both sides learning. What do you think would happen in this case? Would it learn a dierent policy for selecting moves?
Now consider the following scenarios:
(a) Two individual agents play against each other, - left-side-play and right-side-play - each maintaining a separate value-estimate, with both sides learning.
(b) A single agent plays on both the sides using a single value-estimate, with both sides learning. Let this value-estimate correspond to the left-side-play. In order to update these values during right-side-play, the agent flips the current state - changes all "X"s to "O" and vice versa - and updates the values corresponding to this flipped configuration.
Are (a) and (b) identical?
Does (b) converge to some meaningful policy? Does it even make sense?
What is meant by self play? Does it mean scenario (a) or something else?