2
$\begingroup$

I work as a data scientist at a product-oriented company. I guess I will not surprise anyone on this website that it's getting more and more popular for companies with enough data and resources to perform lots of experiments and then make even the smallest decisions based on the results.

Recently, I have been trying to delve into hypothesis testing and A/B tests, in particular, so that I can understand the underlying mathematics, not just use one of the popular calculators in order to calculate the necessary sample size. I have noticed that it's quite a common (and pragmatic) way to get the job done, but I would like to understand every single formula and calculation. I have a Master's degree in applied math and computer science so, as you might understand, just being told a formula doesn't feel right.

One of the most recommended books on A/B testing is this one: https://www.goodreads.com/book/show/51635906-trustworthy-online-controlled-experiments?from_search=true&from_srp=true&qid=IsvcUgzDqk&rank=1

The author is a legend and the book has lots of useful information in terms of methodology, but the mathematical aspect is rather dumbed down. For example, when discussing the sample size, there is a rule of thumb without any derivations that you can use in order to get an estimate of the required sample size.

I have found this website useful, for example: https://online.stat.psu.edu/stat500/

It has several courses with hypothesis testing included, but it's not comprehensive and a lot of derivations are missing as well.

There is the classical textbook on hypothesis testing by Lehman and Romano: https://books.google.co.uz/books?id=Y7vSVW3ebSwC&source=gbs_similarbooks&redir_esc=y

Funnily enough, I find it to be the opposite. It's very rigorous and yet I cannot get rid of the feeling that the authors don't care about the practical side of the theory.

I am wondering if there are any great and modern resources (preferably, books) that strike balance between rigor and being practical. I have heard that it might make sense to look for biostatistics books or trials in medicine in general because hypothesis testing is super relevant in that field, but anyway I don't know any good resources except for those with good reviews on Amazon.

Thanks in advance for any considerations and pieces of advice!

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ I think your problem is that you're comparing introductory level materialto graduate level mathematics texts. The former are limited to algebra, and one or two lines of code. The latter require calculus, even real analysis, and advanced programming skill. What is "practical" to you depends what level you're at. When you're an expert, you need methods to assess the need for, employ, and interpret Bayesian, resampling, non-parametric, etc. statistics. $\endgroup$
    – AdamO
    Commented Apr 19, 2021 at 20:34
  • $\begingroup$ I have nothing against calculus and even real analysis. I took courses in measure theory and functional analysis, but I am definitely not an expert in statistics. Hence the question ;) $\endgroup$
    – Don Draper
    Commented Apr 19, 2021 at 20:36
  • 1
    $\begingroup$ The first book you mention has two other authors, Diane Tang and Ya Xu, who are also prominent practitioners and co-authors on many of the papers that the book covers. $\endgroup$
    – dimitriy
    Commented Apr 21, 2021 at 0:26

1 Answer 1

Reset to default
1
$\begingroup$

I suspect what you are looking for does not really exist since the market for applied rigor is small. Some people care deeply about the convergence of sequences of random numbers, and some just want to test stuff well and make good decisions. The overlap between the two is small.

I would recommend 2-3 books that come close to what you seek:

  1. Statistical Methods in Online A/B Testing by Georgi Georgiev
  2. Field Experiments by Gerber and Green
  3. Running Randomized Experiments by Glennerster and Takavarasha

The first covers some esoteric but relevant topics like one-sided hypothesis tests and confidence intervals, holdouts, and percentage changes standard errors that are ignored or touched on briefly in the conventional treatments. It has an exhaustive list of common misunderstandings of these and many other concepts. The presentation is thoughtful and deep, though not in an excessively mathematical way. I think this has become my go-to book for these topics and a very good complement to textbooks on statistics or field experiments and the Kohavi, Tang, and Xu book that covers more terrain. It has no formula derivations but lots of intuition and some references. It is also somewhat philosophical (especially in the first chapter).

The next two books are by social scientists who do field experiments with people, so many of the same issues arise as in tech (compared to lab experiments in the non-social sciences). The second one is more technical, and the third is geared more towards development economists and is more verbose. Neither one has proofs, but lots of intuition and applications and references.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.