2
$\begingroup$

If an experimenter assigns subjects to an experimental vs a control group truly randomly, is there still any point to performing baseline comparisons between the groups, after they've been so defined?

To me this seems very strange, but I see it done quite often in clinical psychology papers (example, p.4). Were a baseline difference to be found, it seems that either ignoring it or messing around with the group allocation to make the groups "truly" equivalent on whatever measure of interest, would both be dodgy practices. So, should such a check even be made, by (perhaps) thinking of it as a manipulation check?

In a previous question I asked, one contributor very nicely exaplained the role of random assignment:

At the population level, this is, in fact, impossible. That's the value of random assignment. When subjects are randomly assigned to conditions, then the conditions must be drawing on the same population, because the assignment to conditions is guaranteed to be independent of any features of the subjects. Any population difference in outcomes must be causal effects of the conditions themselves, and nothing prior to that.

It seems to me checking for group differences immediately after assigning subjects to a treatment and control groups, defeats the purpose of why we do random allocation to begin with. Am I wrong?

$\endgroup$
2
$\begingroup$

If you do a hypothesis test the null is that there is no difference and the alternative is that the observed difference is so large that it is unlikely to have arisen by chance. But in the case of random allocation you know for sure that the differences have arisen by chance since you randomised them. The only plausible reason for testing is if you believe the randomisation process may have been subverted in some way. You are perfectly correct that many authors routinely do such a test and it is not picked up by referees and editors but that does not, of course, make it right.

Stephen Senn has written about this in an article entitled "Testing for baseline imbalance in clinical trials" available here behind a paywall although maybe available elsewhere.

$\endgroup$
  • 2
    $\begingroup$ So is it still useful for testing for "subversion", if nothing else? $\endgroup$ – user2974951 Feb 6 at 15:14
  • 2
    $\begingroup$ Senn's suggestion to treat these baseline variables as covariates is fine, but I think the thought that these differences cannot be different than chance because they already have arisen by chance is a bit misleading. Not to say that standard null hypothesis significance tests are the appropriate way to treat them, but given the large number of dimensions on which subjects can vary, it is reasonably expected by chance that they will differ substantially on some of them. The question then is how best to describe these differences. $\endgroup$ – Bryan Krause Feb 6 at 15:17
  • $\begingroup$ @BryanKrause you're right - and this is, in fact, the crux of my question. $\endgroup$ – z8080 Feb 6 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.