2
$\begingroup$

Background: let's say I have a website with a big buy button on it and I preformed an A/B testing with the following result:

Version A (control): conversion rate 5%
Version B (variation): conversion rate 8%
Significant level: 5%
Confidence interval: +/- 1%, (8%-5%) = 3% thus the bound is (2%, 4%)
p-value: 0.02
Null hypothesis: the conversion rate of the control group is the same as the conversion rate of the variation.
Alternative hypothesis：the conversion rate of the control group is different from the conversion rate of the variation.

I made up the numbers above but here is how I would calculate these values:

$$ z = \frac{P_{A} - P_{B}} {\sqrt{\frac{P_A(1-P_A)}{n_A} + \frac{P_B(1-P_B)}{n_B}}} $$

$P_A$: the conversion rate for the control
$P_B$: the conversion rate for the variation
$n_A$: the number of visitors who saw the control
$n_B$: the number of visitors who saw the variation

The formula above is based on the independent two-sample t-test:

$$ t= \frac{\bar{X_A} - \bar{X_B}}{\sqrt{\frac{{S_A}^2}{n_A} + {\frac{{S_B}^2}{n_B}}}}$$

Because click conversion rate is essentially a Bernoulli trial, so it follows the Bernoulli distribution.

With Bernoulli distribution, the variance is calculated as $p(1-p)$, thus replacing ${S_A}^2$ with $p(1-p)$ gives the formula for calculating $z$ above.

My interpretation of the p-value:

Assuming there is no difference between the conversion rate of the control and variation, there is only a 2% chance (1 - p-value) that we observe a difference like we have seen. Thus, there are two explanations:

  1. We've only experimented once and we observed something that only happens less than 5% of the time. Theoretically speaking, it is possible to get such extreme data on the first try. If I reject the null hypothesis incorrectly, I am running an extreme low risk of making the type I error.

  2. We've only experimented once and we observed something that only happens less than 5% of the time, so this must not be luck and we should conclude that the difference is real. Thus, the variation group is indeed different from the control group.

My interpretation of the confidence interval：

If we were to repeat the same experiment 100 times (with 100 different group of people testing both versions), then we should expect to see the true difference between the two groups appear in 95 of those experiments.

We think we are the 95%, and if that is true, then the true difference lay somewhere between 2% and 4%.

But there is also a 5% chance that the range (2%, 4%) completely misses the true difference.

We could also be the 5%, such that the range (2%, 4%) does not contain the true difference.

Is my interpretation of the p-value and confidence interval correct?

Improve this question
$\endgroup$
5
  • $\begingroup$ @DeepNorth thx, I added a little bit more to the CI part $\endgroup$
    – Cheng
    Jul 14, 2017 at 1:38
  • 1
    $\begingroup$ I think your interpretation 2 seems right. But you need to tell what kind of statistics you used to perform the test. Then I would say under your $H_0$, the statistic you got would have 5% change which is quite impossible (small probability event) then I will reject the null hypothesis. And the CI interpretation seems correct to me. $\endgroup$
    – Deep North
    Jul 14, 2017 at 1:43
  • $\begingroup$ @DeepNorth I added the statistics part. $\endgroup$
    – Cheng
    Jul 14, 2017 at 2:21
  • $\begingroup$ It should be 5% chance, not change $\endgroup$
    – Deep North
    Jul 14, 2017 at 3:18
  • 2
    $\begingroup$ maybe these answers can help : stats.stackexchange.com/questions/167972/… and stats.stackexchange.com/questions/166323/… $\endgroup$
    – user83346
    Jul 14, 2017 at 13:25

1 Answer 1

Reset to default
2
$\begingroup$

Your last claim is wrong. There is not a 5% chance that your range 2% to 4% misses the mark. It either misses or it does not, 0% or 100%; you just don't know.

The p-value also assumes you met the assumptions of the statistical test you conducted if you are being pedantic.

For claim number 2, you shouldn't ever conclude definitively from one study. But you have evidence to suggest that the difference is there.

Additionally, this decision to claim there is a difference only happens less than 5% of the time (given an $\alpha$ of .05) only if the effect is literally zero.

The p-value is not the probability of making a type-I error. Before a study is conducted, the chosen $\alpha$-level is. After the study is conducted, you either have made or you have not made a type-I error.

I might rewrite 1 thus:

We've only experimented once and we observed something that only happens less than 5% of the time if the null is true. I may be in the 5% of people who will make an error of claiming a difference from the null when there is none. Thus, if I reject the null, I may have made a type-I error.

I might rewrite 2 thus:

We've only experimented once and we observed something that only happens less than 5% of the time if the null is true. Hence, this is evidence to suggest that the null is unlikely to be true. And we may conclude that there is a difference between the control group and variation group that is not due to sampling error.

Improve this answer
$\endgroup$
6
  • $\begingroup$ thx, I fixed the original post. Is my claim #1 correct? $\endgroup$
    – Cheng
    Jul 14, 2017 at 2:27
  • 1
    $\begingroup$ You are not lucky, you are the 5%. You are unlucky, as you have misleading results. You observed something that isn't there, you are the 5% who are bound to make an error. $\endgroup$
    – Heteroskedastic Jim
    Jul 14, 2017 at 2:43
  • $\begingroup$ I revised claim #1 a bit $\endgroup$
    – Cheng
    Jul 14, 2017 at 4:54
  • $\begingroup$ Actually, the p-value is not the risk of making a type 1 error. It is only the probability of observing data assuming nil effect. You either make a type 1 error or you don't, 0% or 100%; you don't know. $\endgroup$
    – Heteroskedastic Jim
    Jul 14, 2017 at 11:24
  • 1
    $\begingroup$ Thanks a lot for the edit! Should I interpret the chance of making type I error as the way of how to interpret confidence interval? Like, if we are going to repeat the experiment 100 times, I will make the type I error 5 times? $\endgroup$
    – Cheng
    Jul 14, 2017 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.