Self Play in Reinforcement Learning

Question

This is regarding the first exercise in Sutton and Barto's book on Reinforcement Learning. I have a few questions about the exercise; I am not looking for an exact solution.

---

Exercise 1.1: Self-Play Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself, with both sides learning. What do you think would happen in this case? Would it learn a dierent policy for selecting moves?

---

Now consider the following scenarios:

(a) Two individual agents play against each other, - left-side-play and right-side-play - each maintaining a separate value-estimate, with both sides learning.

(b) A single agent plays on both the sides using a single value-estimate, with both sides learning. Let this value-estimate correspond to the left-side-play. In order to update these values during right-side-play, the agent flips the current state - changes all "X"s to "O" and vice versa - and updates the values corresponding to this flipped configuration.

Questions

1. Are (a) and (b) identical?

2. Does (b) converge to some meaningful policy? Does it even make sense?

3. What is meant by self play? Does it mean scenario (a) or something else?

Answer 1

I'm a chess player, so I'll use chess in my answer.

1. (a) and (b) are not identical. In (a), you have two agents playing against each other. Their underlying models might not be comparable. Even if they had the same model, their parameters highly likely wouldn't converge simultaneously. This is like matching two different chess engines. In (b), you have an agent playing against itself for as many times as possible. This is the typical definition for self-learning reinforcement learning, and is commonly seen in chess programming.

2. (b) is very common in chess, so it does make sense. In chess, we can map the definitions like this:

   • agent -> chess engine
   • both sides learning -> update piece-square-table (PST) and evaluation parameters. The PST table defines where White pieces and Black pieces should go. Evaluation parameters define how a position should be evaluated statically. For example, rook on the seventh rook, two-bishop advantage, isolated pawns, protected passed-pawn etc.
   • left side play -> white colour
   • right side play -> black colour
   • flips the current state - Minimax
   • changes all X to O - get the alpha-beta evaluation from the last ply and change it's sign
   • updates the values - update parameters based on whether the game is won/draw/loss, and choose appropriate learning rate

3. I have only experience with self-play defined in (b). To my knowledge, nobody has done (a) for chess engine programming.

NeuroChess is a chess engine by reinforcement learning. The engine learns by the final outcome of the game and it does that by playing itself.