In this PyCon Canada tutorial, the author details a simulation process, in lieu of A/B testing, where two respective distributions are sampled from compared for X iterations. You can collect whatever metadata you like, such as:

  • "Is $\theta_1 > \theta_2$?"
  • Or "By how much is $\theta_1 > \theta_2$?"

Note, this tutorial assumed fully Bayesian posterior inference on $\theta_1$ and $\theta_2$.

My question is could this logic be applied from a non-Bayesian perspective. For example, assume that the click through rates (CTR) of two ad campaigns are approximately normally distributed, fit their normal PDFs via MLE, then sample a,b from their respective distributions, compare, and capture the difference for x iterations?

Would this be a viable alternative to NHST?


1 Answer 1


You've invented something very much akin to permutation testing.

This is often done by

  1. Computing the statistic of interest between the true groups.
  2. Randomly shuffling the data into "surrogate" groups.
  3. Recomputing the statistic on those groups.
  4. Repeat steps 2-3 a large number of times (or until you've exhausted all possible permutations of the data)
  5. Compare the true different from Step 1 against the distribution of surrogate differences created in Steps 2-4.

The shuffled data literally implements the null hypothesis--you know there's no difference between the surrogate groups because you've randomly assigned points to them. The differences between different surrogate groups, collected across many repetitions, therefore provide a distribution of your chosen statistic under the mean. A "significant" difference, in the NHST sense, would be way out in the tails of that distribution.

Bootstraping is a related concept, more aimed at assess uncertainty than inference. I found Randomization, Bootstrap and Monte Carlo Methods in Biology by Manley and Alberto to be really helpful (you do not need much biology background to make sense of it).

  • 1
    $\begingroup$ I’ve heard of permutation tests before but the descriptions went over my head. So if I understand correctly, you’d sample a,b from populations A and B, find the difference between a,b and repeat for x iterations, then you’d have this new delta distribution and make sure that 95% of it (or some arbitrary alpha) doesn’t include 0, meaning the delta is significant? Good chance I’m confused! $\endgroup$
    – jbuddy_13
    Dec 19, 2021 at 2:55
  • 1
    $\begingroup$ That's the gist of it! There are some fussy details (one vs two-sided tests, randomizing non-iid data like time series, etc), but something akin to a t-test works just like you've described. $\endgroup$ Dec 20, 2021 at 17:39
  • 2
    $\begingroup$ @jbuddy_13 Your comment sounds more like a bootstrap interval for the difference between means than a permutation test. They are different in that the latter takes samples from a null model. $\endgroup$ Dec 20, 2021 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.