Return to Answer

• Stockfish was running without an opening book or 6-piece Syzygy endgame tablebase.

• Stockfish was running without an opening book or 6-piece Syzygy endgame tablebase. The sample size was insufficient. The Stockfish version was not the latest. Discussion here.

To me, the numbers in the paper are biased or not reproducible.

• Stockfish was running without an opening book or 6-piece Syzygy endgame tablebase.

CONCLUSION

Google has not proven without doubts their methods are superior to Stockfish. Their numbers are superficial and strongly biased to AlphaZero. Their methods are not reproducible by an independent third party. It's still a bit too early to say Deep Learning is a superior method to traditional chess programming.

EDIT (Dec 2017):

There is a new paper from Google Deepmind (https://arxiv.org/pdf/1712.01815.pdf) for deep reinforcement learning in chess. From the abstract, the world number one Stockfish chess engine was "convincingly" defeated. I think this is the most signfiicant achievemnt in computer chess since the 1997 Deep Blue match. I'll update my answer once I read the paper in details.

EDIT (after reading the paper):

I've read the paper thoughtfully. Let's start off with what Google claimed in the paper:

• They defeated Stockfish with Monte-Carlo-Tree-Search + Deep neural networks
• The match was absolutely one-sided, many wins for AlphaZero but none for Stockfish
• They were able to do it in just four hours
• AlphaZero played like a human

Unfortunately, I don't think it's a good journal paper. I'm going to explain with links (so you know I'm not dreaming):

• https://chess.stackexchange.com/questions/19360/how-is-alpha-zero-more-human has my answer on how AlphaZero played like a human

• The match was unfair, strongly biased. I quote Tord Romstad, the original programmer for Stockfish.

https://www.chess.com/news/view/alphazero-reactions-from-top-gms-stockfish-author

The match results by themselves are not particularly meaningful because of the rather strange choice of time controls and Stockfish parameter settings: The games were played at a fixed time of 1 minute/move, which means that Stockfish has no use of its time management heuristics (lot of effort has been put into making Stockfish identify critical points in the game and decide when to spend some extra time on a move; at a fixed time per move, the strength will suffer significantly).

Stockfish couldn't have played the best chess with only a minute per move. The program was not designed for that.

• Stockfish was running on a regular commercial machine, while AlphaZero was on a 4 millions+ TPU machine tuned for AlphaZero. This is a like matching your high-end desktop against a cheap Android phone. Tord wrote:

One is a conventional chess program running on ordinary computers, the other uses fundamentally different techniques and is running on custom designed hardware that is not available for purchase (and would be way out of the budget of ordinary users if it were).

• Google inadvertently gave 64 threads to a 32 core machine for Stockfish. I quote GM Larry Kaufman (world class computer chess expert):

http://talkchess.com/forum/viewtopic.php?p=741987&highlight=#741987

I agree that the test was far from fair; another issue that hurt SF was that it was apparently run on 64 threads on a 32 core machine, but it would play much better running just 32 threads on that machine, since there is almost no SMP benefit to offset the roughly 5 to 3 slowdown. Also the cost ratio was more than I said; I was thinking it was a 64 core machine, but a 32 core machine costs about half what I guessed. So maybe all in all 30 to 1 isn't so bad an estimate. On the other hand I think you underestimate how much it could be further improved.

• Stockfish gave only 1GB hash table. This is a joke... I have a larger hash table for my Stockfish iOS app (Disclaimer: I'm the author) on my iPhone! Tord wrote:

... way too small hash tables for the number of threads ...

1GB hash table is absolutely unacceptable for a match like this. Stockfish would frequently encounter hash collision. It takes CPU cycles to replace old hash entries.

• Stockfish is not designed to run with that many number of threads. In my iOS chess app, only a few threads are used. Tord wrote:

... was playing with far more search threads than has ever received any significant amount of testing ...

To me, the numbers in the paper are biased or not reproducible.

There is a new paper from Google Deepmind (https://arxiv.org/pdf/1712.01815.pdf) for deep reinforcement learning in chess. From the abstract, the world number one Stockfish chess engine was "convincingly" defeated. I think this is the most significant achievement in computer chess since the 1997 Deep Blue match. I'll update my answer once I read the paper in details.