3
$\begingroup$

I am running a multi-variate email campaign with the following results:

Version----------Users------Conversion

Control Group----150--------2%

Variation A--------1000-------2.1%

Variation B--------1000-------2.9%

Variation C--------1000-------1.4%

How do I statistically know which is the winning variation of this campaign? Should I be using chi-squared tests to determine the winner?

Furthermore, is there any standard way of selecting the size of the control group of a sample if I know the sample size? Should the size of the control group be the same as that of the variations or can I keep it as 5% to 20% of the sample regardless of the size of the sample? This is for marketing campaigns I run regularly.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ The standard meaning of a "multivariate" test or analysis is one that involves more than one response. Yours is a standard univariate response situation. See, for instance, material on Analysis of Variance (ANOVA). Incidentally, there is no clear winner here because the control group is too small. $\endgroup$
    – whuber ♦
    Commented Oct 21, 2016 at 12:56
  • $\begingroup$ Alright. 2 follow up questions in this case: (1) Is there a formula to determine the minimum size of a control group? (2) If the Control Group had 500 responses with the same conversion rate, would there still be no clear winner here? $\endgroup$
    – Arpit Rai
    Commented Oct 21, 2016 at 13:03
  • 1
    $\begingroup$ There is an entire branch of statistics, Experimental Design, that (among other things) is concerned with determining optimal sizes of control groups and experimental groups. One insight it offers is that since you are comparing your treatment groups to a control, and the uncertainty in the comparison depends on the uncertainties in both the treatment and the control groups, then you should make the control group size at least as large as any of the treatment groups. If you have a budget for 3150 observations, then the control group should have at least 788 of them. $\endgroup$
    – whuber ♦
    Commented Oct 21, 2016 at 13:13
  • 1
    $\begingroup$ (Continued) However, suppose you obtained the same percentages with group sizes of 1000 each. The standard error of the difference between group B and the control would be 0.69%, making their difference of 0.9% only 1.3 standard errors. Although that is mild evidence that at least one variation exceeded the control, most people would not consider it significant. A simple logistic regression, a chi-squared test, or even an ANOVA would all help you carry out these calculations, but they will agree you don't have enough data to determine whether any variation is better than the control. $\endgroup$
    – whuber ♦
    Commented Oct 21, 2016 at 13:17
  • $\begingroup$ Great. Thanks a ton! This is really really useful! :) I have one last unrelated question if you don't mind. Is there any relation between the size of the sample and the control group I should create out of this sample? Should I always create a control group sized similar to the variations or can I just keep it to 5% to 20% of the sample regardless of sample size? I know you mentioned there is an entire branch dedicated to this but I thought I’d still ask. $\endgroup$
    – Arpit Rai
    Commented Oct 21, 2016 at 13:37

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.