5
$\begingroup$

Recent advance in Google's AlphaGo show a really powerful use of deep learning and reinforcement learning in the complicated space (Go). How did we use deep learning and reinforcement learning together, for example, in Atari or Go?

As far as I know, when we say use them together, we are talking about use deep learning (e.g., CNN) to predict the Q in the reinforcement learning, and then use the Q to make the decision, am I right?

$\endgroup$
  • $\begingroup$ The AlphaGo paper is called Mastering the Game of Go with Deep Neural Networks and Tree Search and Atari RL Playing Atari with Deep Reinforcement Learning. They explain everything very well. $\endgroup$ – Don Reba Mar 16 '16 at 6:03
  • 2
    $\begingroup$ @DonReba I read that paper and did not fully understand it. That's why I posted a question here. $\endgroup$ – RockTheStar Mar 16 '16 at 17:52
4
$\begingroup$

Short Answer:

AlphaGo used reinforcement learning to further tune its policy function deep neural network, which it then used to simulate many games for its value function deep neural network. Collectively, these two deep neural networks were then used to dramatically reduce the space of optimal moves to search, horizontally via the policy network and vertically via the value network.

Long Answer:

AlphaGo built two different deep learning neural networks. The first network predicted which move an expert would make. AlphaGo then used reinforcement learning to further tune this neural network by making it play many games against itself. Both the supervised learning approach and reinforcement approach used back-propagation to update the weights of the neural net. The simulated games were then used to build a second deep learning neural network to predict whether AlphaGo would win the game given the state of the board.

When playing a live game, AlphaGo would then use the first neural network to find likely moves. Promising moves were evaluated in two ways. First AlphaGo would use a simple softmax/logistic regression model to quickly simulate moves (after the promising move) until someone won. They used logistic regression for this simulation instead of the deep net because it could run in microseconds instead of milliseconds (they had to run many simulations). Second, they would use the value function deep neural net to predict who would win. They would then average the result of the predicted win/loss with win/loss of the simulated game to arrive with an estimated value for the promising action. These two evaluation techniques were then repeated many, many times (via a processed called Monte Carlo Tree Search, MCTS) before AlphaGo picked its actual move.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the neat response. I have followup questions for long version: (1) In the first move, the NN predict what the opponent move, and by that they can estimate if he will win or not (which imply a loss there). (2) Does the system only limited to only current (me) and the next (opponent) step? I thought it will looking into more steps. (3) How does RL involve? Is that in the MCTS? $\endgroup$ – RockTheStar Apr 13 '16 at 16:00
  • $\begingroup$ There are separate deep net models for predicting moves and predicting win/loss. Step 1: you predict the move using the policy deep net. Step 2: you predict who will win using the value deep net. The whole process is a bit complicated. In addition to predicting the next move (step 1), and who will win if that move is taken (step 2), there is another part that simulates playing that game to the end until someone wins (step 3). 1-2-3 are essentially repeated many times, and that repetition is called MCTS. $\endgroup$ – Ryan Zotti Apr 13 '16 at 21:33
  • $\begingroup$ My understanding is that reinforcement learning is not used when going live (playing against an actual person). It's just used for strengthening the move prediction network after supervised learning and creating the training dataset for the win/loss prediction networ. $\endgroup$ – Ryan Zotti Apr 13 '16 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.