I am evaluating a Bayes AB Test on 2 variants, A and B.

I then plotted a graph which shows the Probability of B is better than A on a daily basis. My worry comes in on the topic of 'peeking'. Let's say I wanted to find 90% Probability of B is better than A to end the AB Test, I would have ended the test on day 11 as per graph. As you can see, the trend still fluctuates though and also trickles into the opposite direction (lower than 90%) until about day 25 where it seems to start converging.

My question: What are the best practices to ensure that the distribution is indeed converging so I do not end too quickly in the lens of Bayes Inference?

enter image description here

  • $\begingroup$ I have used the SPRT to good effect to detect tiny changes in click-through rates on Web pages. I believe others may be using similar techniques of sequential testing. $\endgroup$ – whuber Feb 13 '20 at 23:46
  • $\begingroup$ Ending the test too early, before convergence, is a genuine concern. In my opinion, the stopping rule should be based on convergence, not the current estimate of the test parameter. To that end, you could use something like the Geweke diagnositc test to decide when to stop. $\endgroup$ – Earlien Feb 13 '20 at 23:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.