Suppose I run an AB test with the goal of increasing the number of users who purchase a product.
After running the test for "long enough" I get that 5% of control users purchase, and 10% of treatment users purchase and I get a p-value of .01. Great! Reject the null hypothesis and conclude treatment is better than control.
I know that p-value of .01 just means P(data|Null Hypothesis) where here, the null hypothesis is that the treatment and control convert at the same rate. This is, of course, very different from the probability of a change actually occurring [which would be 1- P(Null Hypothesis | data)]. We don't have prior information needed to compute this probability, but it may be substantially different from .01.
Now suppose I want to quantify how much better treatment is versus control. The naïve answer that it's twice as good is almost certainly wrong. How about using confidence intervals?
Suppose I calculate a confidence interval of 10% +- 2% for the treatment group at the 95% confidence level. That means we are 95% confident that the true value for this cohort is between 8% and 12%. Similarly, we calculate at a 95% percent confidence level that the control group has conversion rate between 3% and 7%
Now I know it's wrong to say there's a 95% PROBABILITY that the true value lies in this range, but I'm having trouble understanding how the p-value approach relates to the confidence interval approach.
It seems to me that it's inconsistent to claim that on one hand we can't compute the probability that the treatment is better than control (since it requires reversing the conditional probability) and on the other hand claiming that we are 95% confident that the true conversion rate for treatment is between 7% and 13%.
Questions I have:
How does constructing confidence intervals relate to assuming the null hypothesis? In this case, is seems like we're able to state with high confidence that treatment and control are different AND say with high confidence that treatment > control, whereas before, we could not quantify the probability that treatment > control, only whether the discrepancy was inconsistent with the null hypothesis.
If we are 95% confident that treatment users have a conversion rate between 8% and 12% and ALSO are 95% confident that control users have a conversion rate between 3% and 7%, can we make a statement about our confidence of the % increase?
It would be something like with 90.25% confidence (.95^2), the % increase is between 14.3% (8/7-1) and 300% (12/3-1). There should be a better way of doing this (one centered around the observed % increase of 100%)
Thanks for all your help! MB