In the recent nature paper "Mastering the game of Go without human knowledge", how do they incorporate game rules into the convolutional neural network? What is not clear to me is that whether the moves proposed by the convolutional neural network always adhere to the game rule or not.
1
-
$\begingroup$ They did it in the same way as with any other reinforced learning case. The moves need to follow the rules, otherwise the network wouldn't be able to learn anything. How could it learn that moves outside the board are impossible, if they were possible for the algorithm? $\endgroup$– TimCommented Feb 15, 2018 at 16:47
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
In AlphaGo and AlphaGo Zero, one of the raw outputs of one of the neural network is a grid of 19x19 positions with the probability to play a stone in each position - the policy network. Another is an evaluation of any position - the value network. These are filtered by a rule that will only search amongst legal moves, and the probabilities of taking a move are normalised according to the allowed moves.
The agent never learns to only play legal moves. Instead, that restriction is forced on it from the very start by the environment implementing the rules of Go. This is in fact a very common approach in RL. As a result, the network might suggest trying to play illegal moves, if this hard-coded restriction was removed.
It is possible to create and train an agent that will learn to avoid illegal moves, by allowing those actions to occur in the game and having some consequence (such as forfeiting the game, or missing a turn). That could make sense in some environments where failing to complete a game or take a turn might be realistic outcomes.
However, in many card and board game environments, adjusting the action choice space to follow the rules is simpler.
It is also technically possible to teach the agent to avoid suggesting bad moves by assigning a negative reward plus a re-try if it suggests an illegal move. That could be made to work, although the impact would mostly be the occasional extra computation to reject a suggested move during MCTS look-ahead (when the agent incorrectly suggests an illegal move, scores the suggestion negatively and re-tries). There is no reason to suspect that this would improve gameplay over the hard-coded rules approach, although it might be interesting to see how well the agent learns the rules when it has a primary goal of winning.
