Firstly, I'll apologise for my limited understanding on this topic.

Let's say I am running an A/B test with a pool of 20,000 prospects and I split them 50/50 into A and B.

The problem is, the number of people seeing my experiment is skewed toward one group.

Group A has ~7.32% seeing the experiment Group B has ~6.34%

Out of 10,000 people that gives me a pool of

A - 732 people B - 634 people

I'm measuring conversions, so pool A has a conversion rate of 19.8% (~145 people), pool B has a conversion rate of 18% (~114 people)

My instinct says that since pool A has a greater sample size, the result is unreliable. How would I go about eliminating this variation?

is it a p value thing?

  • 1
    $\begingroup$ Just to clarify: Is the initial pool 10K or 20K? Based on your second sentence, it seems that there are 5K per arm but based on sentences 4 to 6 it seems there are 10K per arm. (In general, when we have unequal sample sizes, the power of the test is mostly dependant to the smallest of the two samples.) $\endgroup$ – usεr11852 Nov 28 '18 at 0:35
  • $\begingroup$ You're very right, I'll update the initial pool to 20,000 for simplicity. $\endgroup$ – 3stacks Nov 28 '18 at 0:42
  • 1
    $\begingroup$ OK, cool. So, assuming a response rate of ~6.83% the binomial coefficient are just on the opposite edge of the observed response rate (Hmisc::binconf(n= 10000, x = 683, alpha = 0.05, method = "all")) so I would suggest you take a quick look if there are any hidden variables that might influence the probability of seeing your experiment. (Fun question, I +1ed already!) $\endgroup$ – usεr11852 Nov 28 '18 at 0:46
  • 1
    $\begingroup$ (The above being said, I do not think there are any significant difference in the conversion rates between the two arms.) $\endgroup$ – usεr11852 Nov 28 '18 at 0:59
  • $\begingroup$ Thank you for the info! I'm not that familiar with R, but what you're saying makes sense. You're right, there is a hidden variable preventing even distribution. It's sent out via email so it's pre-determined, whereas we should be doing it on the fly to enforce a 50/50 split. $\endgroup$ – 3stacks Nov 28 '18 at 1:11

Assuming that conversion rate is not affected by participation rate (sample size), here is Minitab's output for your data.

Test and CI for Two Proportions 

Sample    X    N  Sample p
1       145  732  0.198087
2       114  634  0.179811

Difference = p (1) - p (2)
Estimate for difference:  0.0182767
95% CI for difference:  (-0.0232830, 0.0598364)
Test for difference = 0 (vs ≠ 0):  Z = 0.86  P-Value = 0.389

So as @user11852 has Commented, there is not a significant difference between the two conversion rates.

Furthermore, a power analysis shows that, for practical purposes, samples of about these sizes would be nowhere near sufficient to detect a real difference in conversion rates of 19.8% compared with 18%.

Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.18
α = 0.05

              Sample  Target
Comparison p    Size   Power  Actual Power
       0.198    7426    0.80      0.800045
       0.198   12293    0.95      0.950012

The sample size is for each group.

You would need for the group with the smaller sample size to have about 7500 subjects in order to have power 80% for detecting such a small difference in conversion rates.

enter image description here


Sample sizes between your groups can vary without it causing a problem for inference. As others mentioned, an exception is if what causes group assignment is correlated with outcome. The biggest hit for inference is that you'll have lower experimental power as the ratio between sample sizes differs from being equal.


+1 both to Bruce's and Russell's answer, they are constructive. Just to expand (briefly?) on my comment.

We can view the process of a user seeing one of the two versions as a random trial. Mathematically we refer to a single success/failure experiment is a Bernoulli trial, a series of sequence of $N$ independent experiments, each having such a success/failure outcome defines a binomial distribution. Using this statistical notions we can now formalise our problem a bit more rigorously.

Because we have 20,000 users and 1,366 views, we can assume that we can 20K potential experiments and 1,366 successes; that works out as $\sim 6.83\%$ probability of success for our binomial distribution. We want to see now whether the data observed in each branch is likely to come from this super-population of users. Notice that we treat our users as being independent of each other. If they are related (e.g. due to locale), we need to control for this first.

There are a couple of ways of getting a binomial proportion confidence interval. The quickest/dirtiest is to use what is called the "normal approximation", it often also called Wald, or asymptotic interval. i.e. we define: a mean (or expected value), a variance and a standard error based on the available sample size and we hope that the normal approximation works alright. Here, given we have 10K samples probably it will. So we have $\hat{p} = 0.0683$ as our expected value, $\hat{p} (1-\hat{p})$ as our variance, a sample size $n$ of 10K and $\sqrt{ \frac{ \hat{p} (1-\hat{p}) }{n}} = 0.0025226$ as our standard error. Using the normal approximation for a 95% confidence interval (CI) we would get the associate quantile of the normal distribution, here that is $1.96$. Putting all this together we get a confidence interval, of $[0.06335, 0.07324]$ that corresponds directly to $\hat{p} \pm 1.96 \sqrt{ \frac{ \hat{p} (1-\hat{p} )}{n}}$.

In my comment I mentioned that we could the function binconf from the Hmisc package to get this answer directly too. It outputs not only the "normal approximation" CIs but also some other type too. Given that our sample is pretty chunky (10K) they are all very similar.

> Hmisc::binconf(n= 10000, x = 683, alpha = 0.05, method = "all")

#             PointEst      Lower      Upper
# Exact        0.0683 0.06343206 0.07342113
# Wilson       0.0683 0.06351974 0.07341181
# Asymptotic   0.0683 0.06335579 0.07324421 << We calculated this manually!

OK, not that we got these figures, we can look at our observed data and see that our $6.34$ and $7.32$ observed rates for variant A and B respectively, sit firmly on the opposite ends of our CI. Does this spell out trouble? Probably not, but it might mask something fishy. That is why I commented: "take a quick look if there are any hidden variables that might influence the probability of seeing your experiment".

A bit more discussion about your instincts. Having more data is (almost always) a good thing! :) It does not help us to eliminate variation but rather to account/estimate it more accurately. As Bruce has already shown measuring a statistically significant difference between two samples having proportions $\sim 19.8\%$ and $\sim 18.0\%$ respectively, actually takes a quite a few more samples than about 700; we will need roughly 7.5K samples per branch.

The formulas involved are slightly more involved, employing some concepts like pooled proportion and matching ratio that are outside the scope of this answer. A book like: Chow et al.'s "Sample Size Calculations in Clinical Research probably will cover almost all major aspect of sample size calculations if you want to dwell on this further. I am sure, that you will find dozens of Stats books too that have small chapters on this matter too.

For the case examined here, assuming approximately $700$ users per branch, a base conversion rate of $19\%$ and usual $5\%$ significance level and $80\%$ power, we would be confident that our observed difference between the two branches is not due to chance/noise if the new conversion rate was about $24.5\%$.

In R the code would read something like:

> power.prop.test(n = 700, sig.level = 0.05, power = 0.80, p1 = 0.19,
                alternative = "one")

# Two-sample comparison of proportions power calculation 
#          n = 700
#         p1 = 0.19
#         p2 = 0.2447536
#  sig.level = 0.05
#      power = 0.8
# alternative = one.sided
# NOTE: n is number in *each* group

There are quite a few A/B test sample size calculators online (e.g. here, here and here) that if you play around with them you can get a slightly better feel about how many user are necessary for a reasonably reliable A/B test. Notice that most online calculators do not mention if they examine one- or two-side tests (i.e. if we examine experiment we want to detect "larger or smaller than" effects (a two-sided test) or "strictly larger than" (or "smaller than") effects (a one-sided test); this might perplex things a bit.

You might notice that a lot the literature on sample size calculation, comes from Biostatisics/Public Health. That's no coincidence; farmer Bob, testing if "No manure" (branch A) versus "Some manure" (branch B) produces a better crop yield, was the proto-A/B-tester.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.