Textbook on reinforcement learning

Question

I am looking for a textbook/lecture notes in reinforcement learning. I'm fond of the "Introduction to Statistical Learning", but unfortunately they do not cover this topic. I know that a book by Sutton and Barto is a standard reference, and perhaps NDP is also good but they are dated 1997-98, and I was hoping to find a more modern exposition since this field is likely to have quite some development in recent time.

Answer 1

I think Sutton and Barto is still the standard. There are a lot of slide decks and notes from AI classes online, but they typically don't go into too much detail.

Sutton and Barto is a little old, but they are preparing a 2nd edition of their textbook. A draft, dated January 2018, is available here; it's linked from Sutton's webpage, which also has the full text of the first edition.

I would look at this before tackling Kochenderfer et al.'s Decision Making Under Uncertainty. That book has some interesting applications (mostly in aviation) but it moves quickly and bounces around a lot. Szepesvári's Algorithms for Reinforcement Learning is also good, but pithy--it takes about twenty pages to get to $\textrm{TD(}\lambda\textrm{)}$, vs. seven chapers and 150 pages in the newer Sutton and Barto.

Other than that, you might try diving into some papers--the reinforcement learning stuff tends to be pretty accessible.

Answer 2

You might want to check out Algorithms for Reinforcement Learning by Csaba Szepesvári, published in 2010. PDF downloadable from the web site. In my opinion, it is a bit more technical than Sutton and Barto but covers less material.

Answer 3

Here you have some good textbooks/references:

Classic

Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p.

The draft for the second edition is available for free: https://webdocs.cs.ualberta.ca/~sutton/book/the-book.html

Russell/Norvig Chapter 21:

Russell SJ, Norvig P, Davis E. Artificial intelligence: a modern approach. Upper Saddle River, NJ: Prentice Hall; 2010.

More technical

Szepesvári C. Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. 2010;4(1):1–103. http://www.ualberta.ca/~szepesva/RLBook.html

Bertsekas DP. Dynamic Programming and Optimal Control. 4th edition. Belmont, Mass.: Athena Scientific; 2007. 1270 p. Chapter 6, vol 2 is available for free: http://web.mit.edu/dimitrib/www/dpchapter.pdf

For more recent developments

Wiering M, van Otterlo M, editors. Reinforcement Learning. Berlin, Heidelberg: Springer Berlin Heidelberg; 2012 Available from: http://link.springer.com/10.1007/978-3-642-27645-3

Kochenderfer MJ, Amato C, Chowdhary G, How JP, Reynolds HJD, Thornton JR, et al. Decision Making Under Uncertainty: Theory and Application. 1 edition. Cambridge, Massachusetts: The MIT Press; 2015. 352 p.

Multi-agent reinforcement learning

Buşoniu L, Babuška R, Schutter BD. Multi-agent Reinforcement Learning: An Overview. In: Srinivasan D, Jain LC, editors. Innovations in Multi-Agent Systems and Applications - 1 . Springer Berlin Heidelberg; 2010 p. 183–221. Available from: http://link.springer.com/chapter/10.1007/978-3-642-14435-6_7

Schwartz HM. Multi-agent machine learning : a reinforcement approach. Hoboken, New Jersey: Wiley; 2014.

Videos / Courses

I would also suggest David Silver course in YouTube: https://www.youtube.com/playlist?list=PL5X3mDkKaJrL42i_jhE4N-p6E2Ol62Ofa

Answer 4

My favourite lectures notes on reinforcement learning are the ones by Andrew Ng in Stanford's course on ML CS229:

Reiforcment learning notes Stanford CS229

You can also download the lecture videos on iTunes. Or on youtube, they start in the following link:

Lecture 16 CS229