Different conclusions from sample size and significance (AB testing)

Question

I am trying to understand the reasoning behind the A/B testing regarding the sample size and conclusion.

Example 1:

We evaluate the effect of an EE activity on student knowledge using pre and posttests. The mean score on the pretest was 83 out of 100 while the mean score on the posttest was 84. Although you find that the difference in scores is statistically significant (because of a large sample size), the difference is very slight, suggesting that the program did not lead to a meaningful increase in student knowledge.

Example 2:

We are doing A/B testing to detect if a new landing page increased a conversion rate from 10% to 11%. In order to detect this effect size (if it exists), we would to calculate the sample size. We get n = 14744 (1). We run the test for proportion (2) and get a p-value of 9.0282481269354485e-06. The result is statically significant but we can also reason it as due to the large sample size. Nevertheless we conclude that landing page B performs better and we switch to it.

In example 1 we conclude that the program did not lead to a meaningful increase because although there is a statical difference but it is due to a large sample size. In example 2 we concluded that modifications to landing page A (resulted in landing page B) are statistically significant and we switched to the landing page B. But the sample size was also quite large in example 2. So why there are differt conclusions if the outcome is the same?

(1). sms.NormalIndPower().solve_power(sms.proportion_effectsize(0.1, 0.11), power=0.8, alpha=0.05, alternative='two-sided')

(2).

np.random.seed(112)
n = 14751
a_land = np.random.binomial(n, 0.1)
b_land = np.random.binomial(n, 0.11)
print(a_land,b_land)
sm.stats.proportions_ztest([a_land, b_land], [n, n])

Answer 1

The issue here is really about practical significance, rather than statistical significance. In both examples, the results were "statistically significant", but in example 1 the size of the effect (i.e. the difference between the two treatment groups) wasn't really of practical value. A score difference of 1 point is usually inconsequential for most student testing situations, although each situation is different. Depending on the context of this example, a one point difference might be a big deal, but usually a 1 point difference on a test, is not of any practical value. Would you consider revising curriculum and spending all that money to do so to get a 1 point difference? I know I wouldn't.

It seems in the second example, on the other hand, the researchers set out exactly to "detect if a new landing page increased a conversion rate from 10% to 11%." In this case, the researchers thought it was of some practical value to get that extra 1% increase in conversation rate. When they carried out their test, the result was statistically significant, and since a 1% increase was pre-determined to be the "effect size" they were hoping to detect, they conclude the change to landing page B was warranted. I suppose swapping out a webpage for a 1% increase has little cost and is worth it given the returns on conversion (i.e. a positive ROI), whereas swapping out an entire curriculum for a 1% increase in test scores, probably isn't.