1
$\begingroup$

I've been reading a fair bit but have some confusion and would like some help understanding how to determine the required size of a control group in the following situation.

Situation: I am trying to determine whether mailing a person an advertisement affected whether they bought the product advertised.

The people in the mail (test) group and the control group will have similar distributions in their behaviours and attributes prior to the test being run.

Question: If I mail n1 people and would like to measure an effect with statistical significance level p = 0.05 and statistical power B = 1 - \beta = 0.8, what should be the size of the control group, n2?

I would like to know the differences in approaches if the metric is:

  1. A proportion, e.g. number of customers/size of group where size of group is n1 or n2 and the number of customers is the number of people who bought the advertised product in the respective group.
  2. A continuous variable such as the total spend of the group.
  3. A long tailed continuous variable such as the spend per person, for which the median is a preferred measure to the mean.

Confusion: Do I need to estimate a baseline value first (e.g. estimate what I expect the proportion to be for one of groups)? Similarly, do I need to decide on the effect size I'm trying to measure (e.g. before knowing what control size I need, stating "I want to be able to measure a difference in the proportions of 5%"?)

It also seems the method of determining the appropriate control size is married to the type of test being run. If the test being run can be incorporated into your answer that would be very helpful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.