1
$\begingroup$

I have an experiment with 4 conditions: control1, control2, treatment1, and treatment 2. Each has approximately the same number of observations. I want to perform a proportions test to compare the number of successes in each group. To increase the number of observations, and decrease the number of comparisons I run, I'd like to combine the two controls into a single control with 2x the observations. Is this good statistical practice?

$\endgroup$
  • 2
    $\begingroup$ Why are there two control groups in the first place? I would suspect there is a good reason. However without more specifics I don't see how we can properly answer your question. $\endgroup$ – Michael Chernick Apr 14 '17 at 18:54
  • $\begingroup$ Two controls to ensure that the sampling methodology works properly. If the controls are wildly different on some metric of interest, then we know there's a problem. $\endgroup$ – jrubins Apr 14 '17 at 19:14
  • 1
    $\begingroup$ How does the experiment work? Simple randomization to A, B, C or D? Presumably not, but one would need that kind of information to give anything like a sensible answer. $\endgroup$ – Björn Apr 15 '17 at 5:25
  • $\begingroup$ Yep, simple randomized sampling. $\endgroup$ – jrubins Apr 15 '17 at 12:14
1
$\begingroup$

First of all note for the record that it is pointless to test the difference between control1 and control2 since if you randomised them you know for sure that any differences are due to chance unless you have reason to believe that the randomisation process was subverted in some way. I know you did not say you were planning to do that.

Other than that there seems no reason not to combine the two groups and use them as a single control group.

$\endgroup$
  • $\begingroup$ Sure, there should be no difference between control 1 and control 2, but it actually happens quite frequently in web-scale A/B testing that we see differences between two controls. It could happen because randomization failed or it could just be bad luck. I've seen both happen, so it's often wise to have a quick 'gut check' that everything is getting randomized properly. Perhaps a follow up question would be if it's advisable to have a control group 2x the size of any of the treatment groups. $\endgroup$ – jrubins Aug 21 '18 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.