I'm currently working on a (manual) calculation for a bayesian A/B test on logNormal data. I'm currently working with simulated data to increase my understanding.

It's giving me some problems, so I wanted to ask questions. My process:

  • Find prior parameters from user-data predating the experiment
  • Find posterior distributions for control and variant group
  • Look at the difference in posterior samples

The results are: enter image description here

The probability of the B variant outperforming the control A should be: print(np.mean(trace['difference_B_A1']>0))

However, this returns a number around 50% even in an A/A test. It seems to me that I should subtract 50% to get the probability that the variant is actually better, but I'm not sure why.

Can someone explain why this probability is 50% in an A/A test and upwards from 50% in a positive A/B test?

  • $\begingroup$ Why not take logs and work with normal models? $\endgroup$ – Glen_b -Reinstate Monica May 16 '19 at 3:12
  • $\begingroup$ @Glen_b: Is that the normal way to treat this? I'm in favour. $\endgroup$ – Josko de Boer May 16 '19 at 9:04
  • $\begingroup$ It's a pretty common approach, at least. $\endgroup$ – Glen_b -Reinstate Monica May 16 '19 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.