Suppose I run an AB test with the goal of increasing the number of users who purchase a product.

After running the test for "long enough" I get that 5% of control users purchase, and 10% of treatment users purchase and I get a p-value of .01. Great! Reject the null hypothesis and conclude treatment is better than control.

I know that p-value of .01 just means P(data|Null Hypothesis) where here, the null hypothesis is that the treatment and control convert at the same rate. This is, of course, very different from the probability of a change actually occurring [which would be 1- P(Null Hypothesis | data)]. We don't have prior information needed to compute this probability, but it may be substantially different from .01.

Now suppose I want to quantify how much better treatment is versus control. The naïve answer that it's twice as good is almost certainly wrong. How about using confidence intervals?

Suppose I calculate a confidence interval of 10% +- 2% for the treatment group at the 95% confidence level. That means we are 95% confident that the true value for this cohort is between 8% and 12%. Similarly, we calculate at a 95% percent confidence level that the control group has conversion rate between 3% and 7%

Now I know it's wrong to say there's a 95% PROBABILITY that the true value lies in this range, but I'm having trouble understanding how the p-value approach relates to the confidence interval approach.

It seems to me that it's inconsistent to claim that on one hand we can't compute the probability that the treatment is better than control (since it requires reversing the conditional probability) and on the other hand claiming that we are 95% confident that the true conversion rate for treatment is between 7% and 13%.

Questions I have:

  1. How does constructing confidence intervals relate to assuming the null hypothesis? In this case, is seems like we're able to state with high confidence that treatment and control are different AND say with high confidence that treatment > control, whereas before, we could not quantify the probability that treatment > control, only whether the discrepancy was inconsistent with the null hypothesis.

  2. If we are 95% confident that treatment users have a conversion rate between 8% and 12% and ALSO are 95% confident that control users have a conversion rate between 3% and 7%, can we make a statement about our confidence of the % increase?

It would be something like with 90.25% confidence (.95^2), the % increase is between 14.3% (8/7-1) and 300% (12/3-1). There should be a better way of doing this (one centered around the observed % increase of 100%)

Thanks for all your help! MB

  • $\begingroup$ Just to state the obvious because I am uncertain if you appreciate it: The 95% conf. interval you report is a statement about the percentage of conf. intervals that contain the true parameter value (say 10%). It is not that 95% of the future estimates of this experiment will fall in this [8%,12%] range. A conf. interval refers to future conf. intervals, not point estimates. $\endgroup$ – usεr11852 Oct 11 '17 at 21:56
  • $\begingroup$ I get the principle, but I wish I understood it better. I know the statement "there's a 95% probability that the true value lies in the interval" is wrong since the true value is fixed and the interval is random. Can I make some probabilistic statement in an a-priori manner (before I get the results and the confidence intervals)? Either way, I want to understand how to think about tying the confidence intervals to p-values, specifically when it comes to thinking about P(null hypothesis | data). $\endgroup$ – Ben Oct 11 '17 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.