# Different conclusions from sample size and significance (AB testing)

I am trying to understand the reasoning behind the A/B testing regarding the sample size and conclusion.

Example 1:

We evaluate the effect of an EE activity on student knowledge using pre and posttests. The mean score on the pretest was 83 out of 100 while the mean score on the posttest was 84. Although you find that the difference in scores is statistically significant (because of a large sample size), the difference is very slight, suggesting that the program did not lead to a meaningful increase in student knowledge.

Example 2:

We are doing A/B testing to detect if a new landing page increased a conversion rate from 10% to 11%. In order to detect this effect size (if it exists), we would to calculate the sample size. We get n = 14744 (1). We run the test for proportion (2) and get a p-value of 9.0282481269354485e-06. The result is statically significant but we can also reason it as due to the large sample size. Nevertheless we conclude that landing page B performs better and we switch to it.

In example 1 we conclude that the program did not lead to a meaningful increase because although there is a statical difference but it is due to a large sample size. In example 2 we concluded that modifications to landing page A (resulted in landing page B) are statistically significant and we switched to the landing page B. But the sample size was also quite large in example 2. So why there are differt conclusions if the outcome is the same?

(1). sms.NormalIndPower().solve_power(sms.proportion_effectsize(0.1, 0.11), power=0.8, alpha=0.05, alternative='two-sided')

(2).

np.random.seed(112)
n = 14751
a_land = np.random.binomial(n, 0.1)
b_land = np.random.binomial(n, 0.11)
print(a_land,b_land)
sm.stats.proportions_ztest([a_land, b_land], [n, n])

• Is this a homework problem? If so please read the message board rules/tour here stats.stackexchange.com/tour and then add the self-study tag. Also, please tell us what you have done so far or where you might be stuck. Jun 19, 2017 at 1:35
• @StatsStudent no, it is not a homework. That are two examples that I found while reading about A/B Testing and effect size. So, they end up with two different conclusions while the reason is the same. So, I do not understand how it came to different conclusion if the result is the same in both? Jun 19, 2017 at 16:56