AB test not significant in statistical testing, but significant due to sample size? I'm running A/B testing on YouTube thumbnails using proprietary software. After only two days (one day using one thumbnail and another day using the other), my results come back as favoring the new thumbnail with something upward of a 96% confidence level.
However, it's only been two days and the sample sizes are what I would say are small. It seemed odd to me so I ran it really quickly and my p-value was 0.0706 which shows it wasn't significant.
Data for A
Visitors: 24
Conversions: 6
Data for B
Visitors: 29
Conversions: 2
I asked what is causing the discrepancies between previous calculations when I was skeptical and received an answer from a user (the company has been non-responsive to my ticket)

TubeBuddy's A/B testing simply alternates the thumbnails each day
until it reaches statistical significance (if that's the option you
selected). You don't really need a two-sampled test because the sample
size itself determines whether it's significant.
But if you're curious as to whether the sampling that made up the two
groups caused the results to be biased, you could also ways run the
test again to see if you get the same outcome.

I'm still having a hard time understanding how it's significant and what he means about the sample size making it significant when I have such a small sample size. It seems counterintuitive. What am I not getting? (Explain it like I'm five?)
 A: Curious statement: "...[T]he sample size itself determines whether it's significant." A power and sample size computation can give you a clue what sample size is required to detect a particular difference between "thumbnails" A and B, provided that difference is real. However, if there is really no difference between A and B, you can't expect to get a significant result by increasing the sample size.
Using, prop.test in R on the data you show, I get the
results shown below.
prop.test(c(6, 2), c(24, 29))

        2-sample test for equality of proportions 
        with continuity correction

data:  c(6, 2) out of c(24, 29)
X-squared = 2.0942, df = 1, p-value = 0.1479
alternative hypothesis: two.sided
95 percent confidence interval:
  -0.05329726  0.41536622
sample estimates:
    prop 1     prop 2 
0.25000000 0.06896552 

Warning message:
In prop.test(c(6, 2), c(24, 29)) :
  Chi-squared approximation may be incorrect

There are two issues here: (a) My P-value is double
yours, perhaps because I did a two-tailed test. You said nothing
about expecting in advance that the first proportion of conversions would be larger than the second. (b) The
conversion rates are rather small; consequently the
numbers of conversions are too small for a useful
P-value.
In R,prop.test is essentially a chi-squared test
on a $2 \times 2$ table of counts, in which rows are counts for A/B and columns are for conversions Yes/No.  As follows:
a = c(6,18);  b = c(2,27)
TAB = rbind(a,b);  TAB
  [,1] [,2]
a    6   18
b    2   27
chisq.test(TAB)

        Pearson's Chi-squared test 
        with Yates' continuity correction

data:  TAB
X-squared = 2.0942, df = 1, p-value = 0.1479

Warning message:
In chisq.test(TAB) : 
 Chi-squared approximation may be incorrect

The warning message is triggered because some of the
expected counts in the computation of the chi-squared
statistic are smaller than $5.$ Here are the expected counts.
chisq.test(TAB)$exp
      [,1]     [,2]
a 3.622642 20.37736
b 4.377358 24.62264
Warning message:
In chisq.test(TAB) : 
 Chi-squared approximation may be incorrect

As implemented in R, one can use parameter sim=T to obtain
a possibly more useful P-value. However, for your
data the more accurate P-value still does not
indicate a difference between A and B.
chisq.test(TAB, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TAB
X-squared = 3.3582, df = NA, p-value = 0.1149

In summary: (a) There may be a significant difference in
conversion rate between A and B and you don't have
large enough sample sizes to detect that difference.
Alternatively, there may actually be no significant
difference.
Your data shows that you used 53 subjects in
your work. If your data give a useful clue about
the proportions (something like 25% vs 7%), I'd guess that repeating your work
with three or four times as many subjects might give
enough power to find a significant difference.
But no guarantees on that. Here is a Minitab output from one possibly relevant 'power and sample size' computation.
Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.07
α = 0.05

              Sample  Target
Comparison p    Size   Power  Actual Power
        0.25     105    0.95      0.950327
        0.30      72    0.95      0.952430
        0.35      52    0.95      0.950074
The sample size is for each group.

(b) IMHO, there seems to be poor communication between
you and the "user" who tried to help.
(c) If your luck getting useful help from a tech at a software company is no better than mine, I would not
expect an admission anytime soon from them that anything is wrong with their software, its documentation, or their
advice.
A: I think you are doing a questionable statistical test here. What happens on a particular day might be related. E.g. day 1 is a Sunday, so people have a lot of time and click on stuff and convert, day 2 in a Monday and people are more busy and don't. This does not mean that what you did on day 1 is better than what you did on day 2. A sensible analysis here should account for day-to-day variability (i.e. a simple comparison of proportion is probably inappropriate). E.g. a random effects logistic regression with a random day effect on the intercept could be an option.
Randomly assigning one strategy for some users and another for others (on the same day) tends to be better. Alternatively, if you have enough days that you are willing to wait for, randomize days (perhaps in blocks to not have excessively long runs with one approach). Alternating assignment might be okay, but can in some circumstances have problems that randomization doesn't have.
Finally, the company's approach of keeping on going until you have significance invalidates the hypothesis test and if you only go long enough you will always reach significance with this approach even if there's in truth absolutely no difference.
PS: I don't know what you mean by "...upward of 96% significant."
PPS: I also have no idea what the company is talking about with "You don't really need a two-sampled test because the sample size itself determines whether it's significant." Perhaps there's some meaningful thing at the bottom of what they are trying to say, but it does not really make any sense as written.
