I'm currently working on a (manual) calculation for a bayesian A/B test on logNormal data. I'm currently working with simulated data to increase my understanding.
It's giving me some problems, so I wanted to ask questions. My process:
- Find prior parameters from user-data predating the experiment
- Find posterior distributions for control and variant group
- Look at the difference in posterior samples
The probability of the B variant outperforming the control A should be:
However, this returns a number around 50% even in an A/A test. It seems to me that I should subtract 50% to get the probability that the variant is actually better, but I'm not sure why.
Can someone explain why this probability is 50% in an A/A test and upwards from 50% in a positive A/B test?