Let's say I want to perform an AB-Test with the following properties:

  • We assign unique visitors to each group, not the requests (so each trial corresponds to a unique visitor as long as e.g. the cookie is not deleted)
  • We want to measure "items viewed" per unique visitor (side note)

The question now is: How to define the percentage of all trials / unique visitors which should be assigned to the control group A so that ...

  1. A should be as small as possible
  2. A should not be tested significantly different to B according to a suitable test assuming that there is no difference between A and B
  • $\begingroup$ I do not get it. What do you mean by "theoretically, the number of samples in each group will remain fixed." ? If you can distinguish between unique users, then I suggest to randomly assign them to A and B. If you cannot distinguish them, I suggest to assign the non-unique requests to A,B and plainly assume that the ratio of unique visitors to requests is the same for both groups, since this variable is independent of the stuff you want to test, isn't it ? $\endgroup$
    – mlwida
    Jan 17 '12 at 7:30
  • $\begingroup$ Second question regarding "If we define ... without even applying any "treatment": What is this percentage you define ? Control ? So you have 4 groups in total ? How do you define this group ? Random assignment ? Some numbers i.e. a specific example would be very helpful. Looks like an interesting question though. $\endgroup$
    – mlwida
    Jan 17 '12 at 7:32
  • $\begingroup$ @steffen 1st question: the number of unique users that visited the website is almost the same. Each user is randomly assigned to a group (A or B) and this do not change (only if the cookie is deleted). I'm trying to measure some statistics like the number of items visited per unique user. $\endgroup$
    – zeferino
    Jan 17 '12 at 9:46
  • $\begingroup$ @steffen 2nd question: I have two groups (A and B) and each unique user is assigned to a group randomly according to a percentage. My question is how to define this percentage in a manner that I garantee that this two groups remains statistically equal (considering the statistic of interest) and that group A has the lowest number of users possible without applying any treatment yet. If one group has a very low number of users, the measurement of the effect of any treatment may be affected. $\endgroup$
    – zeferino
    Jan 17 '12 at 9:51
  • 1
    $\begingroup$ I have reworded the question according to your comments. Please check if I have accidentically changed the meaning. $\endgroup$
    – mlwida
    Jan 17 '12 at 16:38

By saying

A should not be tested significantly different to B according to a suitable test assuming that there is no difference between A and B

you mean that you want to keep the so called Type I-error, i.e. rejecting the Null-Hypothesis although it is true, as small as possible. This can be simply achieved by setting $\alpha$ of the statistical test to the minimum accepted error probability (> 0).

However, this is not enough. What you actually want to do is to control both type of errors, or otherwise you won't detect a difference although it exists. In general, one calculates the required sample size given a minimum difference to detect (controlling the Type II-error, i.e. not rejecting the Null-Hypothesis although it is false) and a maximum accept $\alpha$ and $\beta$. $\beta$ is also called the "power" of a statistical test, hence another keyword to look for is "power analysis" or "power calculation".

Every basic textbook about statistics should contain a chapter about that. Regarding application I suggest to use the r-package "pwr" which already provides implementations for the most common cases.

Example: Assuming that the distribution of A and B is both approximately normal distributed and that the variances are known to be equal, this method pwr.t2n.test will help.

percentageA <- seq(0.1,0.9,by=0.05)
n <- 1000 
beta <- 0.8
difference_2_detect <- 0.1
alternative ="greater"

y <- as.numeric(lapply(percentageA,function(x){
    pwr.t2n.test(n1 = ceiling(n*x), n2=n-ceiling(n*x), 
            d = difference_2_detect,sig.level=NULL, power = beta,
            alternative =alternative)$sig.level

plot(percentageA,y,xlab="percentage for A",ylab="alpha")

results in the following plot, which not surprisingly shows that the best split is at 0.5.

enter image description here

Why ? Because a) the variances are equal and b) to make the best statement one needs to add more samples to both groups in order to shrink both confidence intervals. If only the confidence interval of one group is made smaller, the other one will remain large and one cannot reject the Null-Hypothesis (at least not with a small alpha). Note that the confidence intervals shrink by factor $\sqrt(n)$, hence different percentages result in different alphas.

I know that keeping control groups is often expensive. Either the control group gets more data at once (equal split) or the test lasts longer. In the latter case, the control group will require a lesser amount of data overall, just because the confidenceinterval of B has shrunk more.

Note: I admit, I do not know whether there is a formula for calculating the required sample size where

  • the variances are unknown (and especially it is unknown whether they are equivalent)
  • the sample sizes differ

If you require that, you may ask specifically for it in a seperate question. My knowledge ends here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.