I am trying to understand the reasoning behind the A/B testing regarding the sample size and conclusion.
Example 1:
We evaluate the effect of an EE activity on student knowledge using pre and posttests. The mean score on the pretest was 83 out of 100 while the mean score on the posttest was 84. Although you find that the difference in scores is statistically significant (because of a large sample size), the difference is very slight, suggesting that the program did not lead to a meaningful increase in student knowledge.
Example 2:
We are doing A/B testing to detect if a new landing page increased a conversion rate from 10% to 11%. In order to detect this effect size (if it exists), we would to calculate the sample size. We get n = 14744 (1). We run the test for proportion (2) and get a p-value of 9.0282481269354485e-06. The result is statically significant but we can also reason it as due to the large sample size. Nevertheless we conclude that landing page B performs better and we switch to it.
In example 1 we conclude that the program did not lead to a meaningful increase because although there is a statical difference but it is due to a large sample size. In example 2 we concluded that modifications to landing page A (resulted in landing page B) are statistically significant and we switched to the landing page B. But the sample size was also quite large in example 2. So why there are differt conclusions if the outcome is the same?
(1). sms.NormalIndPower().solve_power(sms.proportion_effectsize(0.1, 0.11), power=0.8, alpha=0.05, alternative='two-sided')
(2).
np.random.seed(112)
n = 14751
a_land = np.random.binomial(n, 0.1)
b_land = np.random.binomial(n, 0.11)
print(a_land,b_land)
sm.stats.proportions_ztest([a_land, b_land], [n, n])