3
$\begingroup$

I have a data set where the control group sample size is significantly smaller than the experimental (4-5x in my situation, but I'm interested in the general answer). A colleague was trying to convince me to randomly filter my experimental data down to the same size as my control, but it seems to me that more data is always better.

I should be using all of the data available to me to shrink my error bars as much as possible for means testing, correct? Increasing the variability of one group by artificially decreasing the sample size seems to be counter productive.

Is there any merit to my colleague's suggestion? Does it matter if my control or experimental group is the smaller one?

$\endgroup$
1
$\begingroup$

I don't think your colleague is correct in this, especially not given that the data has been collected already. Arbitrarily shrinking sample size seems, as you mention, counter-productive at best.

In some circumstances it might not be. For example, I once performed several simulated nested studies within a larger cohort study, to illustrate the decrease in bias and increase in precision that comes from adding more controls for a class. The figures are as follows, for a 1:1 Case:Control ratio, a 3:1, 5:1, 8:1, 10:1:

enter image description here

enter image description here

In both cases, you can see that adding more controls (as seen by the increasing cost) results in both a less biased and more precise answer, but the relative gain drops off pretty swiftly. This might make the case for not collecting more data, but given you already have it, the difference in cost is zero.

$\endgroup$
  • $\begingroup$ Right, I've already collected the data so it doesn't make sense to throw it away. I see what you're saying from a DOE perspective though. Thanks for the reality check! $\endgroup$ – Andrew Sep 8 '16 at 0:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.