# AB test not significant in statistical testing, but significant due to sample size?

I'm running A/B testing on YouTube thumbnails using proprietary software. After only two days (one day using one thumbnail and another day using the other), my results come back as favoring the new thumbnail with something upward of a 96% confidence level.

However, it's only been two days and the sample sizes are what I would say are small. It seemed odd to me so I ran it really quickly and my p-value was 0.0706 which shows it wasn't significant.

Data for A

Visitors: 24

Conversions: 6

Data for B

Visitors: 29

Conversions: 2

I asked what is causing the discrepancies between previous calculations when I was skeptical and received an answer from a user (the company has been non-responsive to my ticket)

TubeBuddy's A/B testing simply alternates the thumbnails each day until it reaches statistical significance (if that's the option you selected). You don't really need a two-sampled test because the sample size itself determines whether it's significant.

But if you're curious as to whether the sampling that made up the two groups caused the results to be biased, you could also ways run the test again to see if you get the same outcome.

I'm still having a hard time understanding how it's significant and what he means about the sample size making it significant when I have such a small sample size. It seems counterintuitive. What am I not getting? (Explain it like I'm five?)

• I'm not sure I know what you mean by "...upward of 96% significant." Commented Jun 1, 2022 at 13:08

Curious statement: "...[T]he sample size itself determines whether it's significant." A power and sample size computation can give you a clue what sample size is required to detect a particular difference between "thumbnails" A and B, provided that difference is real. However, if there is really no difference between A and B, you can't expect to get a significant result by increasing the sample size.

Using, prop.test in R on the data you show, I get the results shown below.

prop.test(c(6, 2), c(24, 29))

2-sample test for equality of proportions
with continuity correction

data:  c(6, 2) out of c(24, 29)
X-squared = 2.0942, df = 1, p-value = 0.1479
alternative hypothesis: two.sided
95 percent confidence interval:
-0.05329726  0.41536622
sample estimates:
prop 1     prop 2
0.25000000 0.06896552

Warning message:
In prop.test(c(6, 2), c(24, 29)) :
Chi-squared approximation may be incorrect


There are two issues here: (a) My P-value is double yours, perhaps because I did a two-tailed test. You said nothing about expecting in advance that the first proportion of conversions would be larger than the second. (b) The conversion rates are rather small; consequently the numbers of conversions are too small for a useful P-value.

In R,prop.test is essentially a chi-squared test on a $$2 \times 2$$ table of counts, in which rows are counts for A/B and columns are for conversions Yes/No. As follows:

a = c(6,18);  b = c(2,27)
TAB = rbind(a,b);  TAB
[,1] [,2]
a    6   18
b    2   27
chisq.test(TAB)

Pearson's Chi-squared test
with Yates' continuity correction

data:  TAB
X-squared = 2.0942, df = 1, p-value = 0.1479

Warning message:
In chisq.test(TAB) :
Chi-squared approximation may be incorrect


The warning message is triggered because some of the expected counts in the computation of the chi-squared statistic are smaller than $$5.$$ Here are the expected counts.

chisq.test(TAB)\$exp
[,1]     [,2]
a 3.622642 20.37736
b 4.377358 24.62264
Warning message:
In chisq.test(TAB) :
Chi-squared approximation may be incorrect


As implemented in R, one can use parameter sim=T to obtain a possibly more useful P-value. However, for your data the more accurate P-value still does not indicate a difference between A and B.

chisq.test(TAB, sim=T)

Pearson's Chi-squared test
with simulated p-value
(based on 2000 replicates)

data:  TAB
X-squared = 3.3582, df = NA, p-value = 0.1149


In summary: (a) There may be a significant difference in conversion rate between A and B and you don't have large enough sample sizes to detect that difference. Alternatively, there may actually be no significant difference.

Your data shows that you used 53 subjects in your work. If your data give a useful clue about the proportions (something like 25% vs 7%), I'd guess that repeating your work with three or four times as many subjects might give enough power to find a significant difference. But no guarantees on that. Here is a Minitab output from one possibly relevant 'power and sample size' computation.

Power and Sample Size

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.07
α = 0.05

Sample  Target
Comparison p    Size   Power  Actual Power
0.25     105    0.95      0.950327
0.30      72    0.95      0.952430
0.35      52    0.95      0.950074
The sample size is for each group.


(b) IMHO, there seems to be poor communication between you and the "user" who tried to help.

(c) If your luck getting useful help from a tech at a software company is no better than mine, I would not expect an admission anytime soon from them that anything is wrong with their software, its documentation, or their advice.

• What do you mean by "There may be a significant difference in conversion rate between A and B and you don't have large enough sample sizes to detect that difference. Alternatively, there may actually be no significant difference." Do you mean "of practically relevant size" when you say "significant"? Commented Jun 1, 2022 at 13:08
• Meant what I said. Commented Jun 2, 2022 at 12:57
• A true parameter value cannot be statistically significant or statistically non-significant, so do you mean "significant" in the non-statistical "of practically relevant size" meaning? If you meant "statistically significant", what you said does not seem to make sense. Commented Jun 2, 2022 at 12:59

I think you are doing a questionable statistical test here. What happens on a particular day might be related. E.g. day 1 is a Sunday, so people have a lot of time and click on stuff and convert, day 2 in a Monday and people are more busy and don't. This does not mean that what you did on day 1 is better than what you did on day 2. A sensible analysis here should account for day-to-day variability (i.e. a simple comparison of proportion is probably inappropriate). E.g. a random effects logistic regression with a random day effect on the intercept could be an option.

Randomly assigning one strategy for some users and another for others (on the same day) tends to be better. Alternatively, if you have enough days that you are willing to wait for, randomize days (perhaps in blocks to not have excessively long runs with one approach). Alternating assignment might be okay, but can in some circumstances have problems that randomization doesn't have.

Finally, the company's approach of keeping on going until you have significance invalidates the hypothesis test and if you only go long enough you will always reach significance with this approach even if there's in truth absolutely no difference.

PS: I don't know what you mean by "...upward of 96% significant." PPS: I also have no idea what the company is talking about with "You don't really need a two-sampled test because the sample size itself determines whether it's significant." Perhaps there's some meaningful thing at the bottom of what they are trying to say, but it does not really make any sense as written.

• Edited. I meant to say It’s at a 96 point something confidence level Commented Jun 1, 2022 at 14:08
• You mean a 96% confidence interval excluded a difference is 0? Commented Jun 1, 2022 at 14:16
• Interesting comment. Somewhat short of an answer. Commented Jun 2, 2022 at 12:58
• @Björn I’m sorry, I’m very new at this. Commented Jun 2, 2022 at 19:50
• @BruceET I’m sorry, I’m very new at this. Commented Jun 2, 2022 at 19:51