I am trying to understand the reasoning behind the A/B testing regarding the sample size and conclusion.

Example 1:

We evaluate the effect of an EE activity on student knowledge using pre and posttests. The mean score on the pretest was 83 out of 100 while the mean score on the posttest was 84. Although you find that the difference in scores is statistically significant (because of a large sample size), the difference is very slight, suggesting that the program did not lead to a meaningful increase in student knowledge.

Example 2:

We are doing A/B testing to detect if a new landing page increased a conversion rate from 10% to 11%. In order to detect this effect size (if it exists), we would to calculate the sample size. We get n = 14744 (1). We run the test for proportion (2) and get a p-value of 9.0282481269354485e-06. The result is statically significant but we can also reason it as due to the large sample size. Nevertheless we conclude that landing page B performs better and we switch to it.

In example 1 we conclude that the program did not lead to a meaningful increase because although there is a statical difference but it is due to a large sample size. In example 2 we concluded that modifications to landing page A (resulted in landing page B) are statistically significant and we switched to the landing page B. But the sample size was also quite large in example 2. So why there are differt conclusions if the outcome is the same?

(1). sms.NormalIndPower().solve_power(sms.proportion_effectsize(0.1, 0.11), power=0.8, alpha=0.05, alternative='two-sided')


n = 14751
a_land = np.random.binomial(n, 0.1)
b_land = np.random.binomial(n, 0.11)
sm.stats.proportions_ztest([a_land, b_land], [n, n])
  • $\begingroup$ Is this a homework problem? If so please read the message board rules/tour here stats.stackexchange.com/tour and then add the self-study tag. Also, please tell us what you have done so far or where you might be stuck. $\endgroup$ Jun 19, 2017 at 1:35
  • $\begingroup$ @StatsStudent no, it is not a homework. That are two examples that I found while reading about A/B Testing and effect size. So, they end up with two different conclusions while the reason is the same. So, I do not understand how it came to different conclusion if the result is the same in both? $\endgroup$
    – Alina
    Jun 19, 2017 at 16:56

1 Answer 1


The issue here is really about practical significance, rather than statistical significance. In both examples, the results were "statistically significant", but in example 1 the size of the effect (i.e. the difference between the two treatment groups) wasn't really of practical value. A score difference of 1 point is usually inconsequential for most student testing situations, although each situation is different. Depending on the context of this example, a one point difference might be a big deal, but usually a 1 point difference on a test, is not of any practical value. Would you consider revising curriculum and spending all that money to do so to get a 1 point difference? I know I wouldn't.

It seems in the second example, on the other hand, the researchers set out exactly to "detect if a new landing page increased a conversion rate from 10% to 11%." In this case, the researchers thought it was of some practical value to get that extra 1% increase in conversation rate. When they carried out their test, the result was statistically significant, and since a 1% increase was pre-determined to be the "effect size" they were hoping to detect, they conclude the change to landing page B was warranted. I suppose swapping out a webpage for a 1% increase has little cost and is worth it given the returns on conversion (i.e. a positive ROI), whereas swapping out an entire curriculum for a 1% increase in test scores, probably isn't.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.