How can I learn statistical a/b testing? I need to learn statistical a/b testing. I have machine learning and probability background, how can I learn about this topic? Which books should I read? Or maybe someone has article suggestions?
 A: Two books come to mind:

*

*Trustworthy Online Controlled Experiments by Kohavi, Tang, and
Xu

*Statistical Methods in Online A/B Testing by Georgi Georgiev

The first book collects a great deal of wisdom distilled from years of experience that was scattered in obscure field journals and touched only briefly in textbooks on statistics and field experimentation. There is also a treasure trove of practical examples and institutional details about online experimentation that is missing from those sources. It covers a great deal of terrain but includes references for those who want to go a bit deeper.
The second book covers some esoteric, but relevant topics like one-sided hypothesis tests and confidence intervals, holdouts, and percentage changes standard errors, that are ignored or touched on briefly in the conventional treatments. It has an exhaustive list of common misunderstandings of these and many other concepts. The presentation is thoughtful and deep, though not in an excessively mathematical way. I think this has become my go-to book for these topics and a very good complement to textbooks on statistics and field experiments and as well as the Kohavi, Tang, and Xu book.
These are both applied books, but they are not exactly cookbooks for how to analyze experimental data. If you lack a background in applied statistics, you may need to supplement with that to implement their advice.
