# Textbook on reinforcement learning

I am looking for a textbook/lecture notes in reinforcement learning. I'm fond of the "Introduction to Statistical Learning", but unfortunately they do not cover this topic. I know that a book by Sutton and Barto is a standard reference, and perhaps NDP is also good but they are dated 1997-98, and I was hoping to find a more modern exposition since this field is likely to have quite some development in recent time.

I think Sutton and Barto is still the standard. There are a lot of slide decks and notes from AI classes online, but they typically don't go into too much detail.

Sutton and Barto is a little old, but they are preparing a 2nd edition of their textbook. A draft, dated January 2018, is available here; it's linked from Sutton's webpage, which also has the full text of the first edition.

I would look at this before tackling Kochenderfer et al.'s Decision Making Under Uncertainty. That book has some interesting applications (mostly in aviation) but it moves quickly and bounces around a lot. Szepesvári's Algorithms for Reinforcement Learning is also good, but pithy--it takes about twenty pages to get to $\textrm{TD(}\lambda\textrm{)}$, vs. seven chapers and 150 pages in the newer Sutton and Barto.

Other than that, you might try diving into some papers--the reinforcement learning stuff tends to be pretty accessible.

• Thanks, I've taken a look at the new edition, but I would not say it is much updated. I'm still interested in a more up-to date exposure. – Ulysses Dec 24 '14 at 11:22
• Yeah, it's definitely not a complete overhaul, but nothing else really comes to mind besides some volumes of Springer's "Lecture Notes", which are essentially just collections of papers. If you find something else, please post an update; I'd love to check it out. – Matt Krause Dec 26 '14 at 22:34
• I see, sure I'll do – Ulysses Dec 29 '14 at 11:40
• @CharlieParker, I'm not sure. The most recent draft (19 June 2017) looks fairly complete and mentions MIT Press, but the MIT Press site appears to be selling the first edition still. For what it's worth, the draft is directly from the authors' public website, so there's no need to be concerned about using a "leaked" version or anything like that. – Matt Krause Jul 26 '17 at 20:10
• @Thomas, I updated the link with a newer draft. – Matt Krause Feb 26 '18 at 15:52

You might want to check out Algorithms for Reinforcement Learning by Csaba Szepesvári, published in 2010. PDF downloadable from the web site. In my opinion, it is a bit more technical than Sutton and Barto but covers less material.

Here you have some good textbooks/references:

Classic

Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Mass: A Bradford Book; 1998. 322 p.

Russell/Norvig Chapter 21:

Russell SJ, Norvig P, Davis E. Artificial intelligence: a modern approach. Upper Saddle River, NJ: Prentice Hall; 2010.

More technical

Szepesvári C. Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. 2010;4(1):1–103. http://www.ualberta.ca/~szepesva/RLBook.html

Bertsekas DP. Dynamic Programming and Optimal Control. 4th edition. Belmont, Mass.: Athena Scientific; 2007. 1270 p. Chapter 6, vol 2 is available for free: http://web.mit.edu/dimitrib/www/dpchapter.pdf

For more recent developments

Wiering M, van Otterlo M, editors. Reinforcement Learning. Berlin, Heidelberg: Springer Berlin Heidelberg; 2012 Available from: http://link.springer.com/10.1007/978-3-642-27645-3

Kochenderfer MJ, Amato C, Chowdhary G, How JP, Reynolds HJD, Thornton JR, et al. Decision Making Under Uncertainty: Theory and Application. 1 edition. Cambridge, Massachusetts: The MIT Press; 2015. 352 p.

Multi-agent reinforcement learning

Buşoniu L, Babuška R, Schutter BD. Multi-agent Reinforcement Learning: An Overview. In: Srinivasan D, Jain LC, editors. Innovations in Multi-Agent Systems and Applications - 1 . Springer Berlin Heidelberg; 2010 p. 183–221. Available from: http://link.springer.com/chapter/10.1007/978-3-642-14435-6_7

Schwartz HM. Multi-agent machine learning : a reinforcement approach. Hoboken, New Jersey: Wiley; 2014.

Videos / Courses