1
$\begingroup$

I have an experiment with 4 conditions: control1, control2, treatment1, and treatment 2. Each has approximately the same number of observations. I want to perform a proportions test to compare the number of successes in each group. To increase the number of observations, and decrease the number of comparisons I run, I'd like to combine the two controls into a single control with 2x the observations. Is this good statistical practice?

$\endgroup$
4
  • 2
    $\begingroup$ Why are there two control groups in the first place? I would suspect there is a good reason. However without more specifics I don't see how we can properly answer your question. $\endgroup$ Apr 14, 2017 at 18:54
  • $\begingroup$ Two controls to ensure that the sampling methodology works properly. If the controls are wildly different on some metric of interest, then we know there's a problem. $\endgroup$
    – jrubins
    Apr 14, 2017 at 19:14
  • 1
    $\begingroup$ How does the experiment work? Simple randomization to A, B, C or D? Presumably not, but one would need that kind of information to give anything like a sensible answer. $\endgroup$
    – Björn
    Apr 15, 2017 at 5:25
  • $\begingroup$ Yep, simple randomized sampling. $\endgroup$
    – jrubins
    Apr 15, 2017 at 12:14

1 Answer 1

1
$\begingroup$

First of all note for the record that it is pointless to test the difference between control1 and control2 since if you randomised them you know for sure that any differences are due to chance unless you have reason to believe that the randomisation process was subverted in some way. I know you did not say you were planning to do that.

Other than that there seems no reason not to combine the two groups and use them as a single control group.

$\endgroup$
1
  • $\begingroup$ Sure, there should be no difference between control 1 and control 2, but it actually happens quite frequently in web-scale A/B testing that we see differences between two controls. It could happen because randomization failed or it could just be bad luck. I've seen both happen, so it's often wise to have a quick 'gut check' that everything is getting randomized properly. Perhaps a follow up question would be if it's advisable to have a control group 2x the size of any of the treatment groups. $\endgroup$
    – jrubins
    Aug 21, 2018 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.