Background: let's say I have a website with a big buy button on it and I preformed an A/B testing with the following result:
Version A (control): conversion rate 5%
Version B (variation): conversion rate 8%
Significant level: 5%
Confidence interval: +/- 1%, (8%-5%) = 3% thus the bound is (2%, 4%)
p-value: 0.02
Null hypothesis: the conversion rate of the control group is the same as the conversion rate of the variation.
Alternative hypothesis:the conversion rate of the control group is different from the conversion rate of the variation.
I made up the numbers above but here is how I would calculate these values:
$$ z = \frac{P_{A} - P_{B}} {\sqrt{\frac{P_A(1-P_A)}{n_A} + \frac{P_B(1-P_B)}{n_B}}} $$
$P_A$: the conversion rate for the control
$P_B$: the conversion rate for the variation
$n_A$: the number of visitors who saw the control
$n_B$: the number of visitors who saw the variation
The formula above is based on the independent two-sample t-test:
$$ t= \frac{\bar{X_A} - \bar{X_B}}{\sqrt{\frac{{S_A}^2}{n_A} + {\frac{{S_B}^2}{n_B}}}}$$
Because click conversion rate is essentially a Bernoulli trial, so it follows the Bernoulli distribution.
With Bernoulli distribution, the variance is calculated as $p(1-p)$, thus replacing ${S_A}^2$ with $p(1-p)$ gives the formula for calculating $z$ above.
My interpretation of the p-value:
Assuming there is no difference between the conversion rate of the control and variation, there is only a 2% chance (1 - p-value) that we observe a difference like we have seen. Thus, there are two explanations:
We've only experimented once and we observed something that only happens less than 5% of the time. Theoretically speaking, it is possible to get such extreme data on the first try. If I reject the null hypothesis incorrectly, I am running an extreme low risk of making the type I error.
We've only experimented once and we observed something that only happens less than 5% of the time, so this must not be luck and we should conclude that the difference is real. Thus, the variation group is indeed different from the control group.
My interpretation of the confidence interval:
If we were to repeat the same experiment 100 times (with 100 different group of people testing both versions), then we should expect to see the true difference between the two groups appear in 95 of those experiments.
We think we are the 95%, and if that is true, then the true difference lay somewhere between 2% and 4%.
But there is also a 5% chance that the range (2%, 4%) completely misses the true difference.
We could also be the 5%, such that the range (2%, 4%) does not contain the true difference.
Is my interpretation of the p-value and confidence interval correct?