How to check if randomization was proper

Question

Randomized controlled trial (RCT), where participants are randomly allocated to 2 or more groups and given different treatments or interventions, to evaluate which one is better, are commonly performed.

For these RCTs, it is now recommended that hypothesis testing for differences in baseline parameters need not be done, since any differences will be due to chance alone.

1. However, there could be various reasons due to which randomization process may fail or may not have been proper. What steps should be taken to see that this is not the case before analyzing data from any RCT.

2. At the same time, it is recommended that baseline parameters should be checked to see "if effect sizes are large and meaningful" (see here). However, this is often not done in reports of RCTs. Should this not be emphasized as an essential step?

To clarify, we are talking about studies done by others which we have to review. If it is not possible to test adequacy of randomization after study is done, should we not perform hypothesis testing on key baseline parameters to find if there are any important differences between the groups?

Thanks for your insight.

Answer 1

The best way to generate a "reproducible randomisation" is to use a scripted randomisation algorithm in a statistical programming language and make sure you "set the seed" for the randomisation. For example, suppose you want a reproducible randomisation in R where you select $t=40$ random treatment objects from a larger group of $n=100$ objects. This could be implemented with code like this.

#Generate randomisation
#Seed was determined from a call to runif(1)*.Machine$integer.max
set.seed(1825154847)
t    <- 40
n    <- 100
RAND <- sort(sample.int(n, size = t, replace = FALSE))

#Show the values of the treatment group
RAND

[1]   6   9  16  18  20  22  23  28  30  31  32  35  39  40  42  43  44  45  47  50
     53  64  67  68  71  72  73  75  79  82  83  84  85  86  90  93  94  96  99 100

As to how you know that this is a "proper randomisation", this hinges on two things: (1) your trust in the underlying pseudo-random number generator in the software you are using; and (2) your trust that the seed was determined independently of the data. With regard to the first issue, the sampling methods in statistical languages like R are built on pseudo-random number generators that have to pass suites of "randomisation tests". These tests measure the outputs of the PRNG against various statistical tests to ensure that there is no significant evidence of non-random patterns. This is usually sufficient to give the user confidence that there are no systematic patterns in the PRNG that will bias the RCT. With regard to the second issue, this is largely a matter of trusting the person who wrote the test, though if this is an issue then it is possible to impose further protocols for choosing the seed (e.g., using a true random number generator with independent witnesses).

With regard to your second point I am not sure what you mean, but I certainly agree with the view that post hoc tests of randomisatoin are discouraged and should not be used. When engaging in statistical hypothesis tests for "effects" of a treatment, the usual caveats on results should be borne in mind --- particularly the difference between "statistically significance" (i.e., significant evidence of a non-zero effect size), versus "practical significance" (i.e., substantial magnitude of the effect size). The randomisation in the RCT is a protocol to remove causal pathways from would-be confounding variables to the treatments, to assist in making causal inferences from the statistical inferences. It does not change the normal caution required for the interpretion of the statistical aspects of the inferences.

Finally, it is important to bear in mind that if one wishes to randomise treatments, but still "balance" some covariates, this can be done using block randomisation. This is the proper randomisation method to use if you want to balance some covariate while still desirable randomisation properties --- it is far superior to doing a simple-random-sample and then performing post hoc checks for balance. For example, if you have an overall group that is equally composed of males and females, and you want a treatment and control group to also have equal numbers of males and females, you can do this by forming "blocks" of male-female pairs and then randomly allocating to the groups from these blocks. (A full description of the process is beyond the scope of this answer, but look it up if you are interested.)

Answer 2

If you are running the experiment

If you're concerned that the randomization might be implemented incorrectly, it is generally more effective to take care at the front end to ensure the randomization is done right, than to try to detect it at the back end. There are several reasons for this. First, detecting that the randomization failed is difficult, error-prone, and ambiguous. Second, even if you obtained evidence that the randomization failed, what would you do? It's unclear and anything you did at that point would be on shaky ground.

It's probably better to put your energy into reviewing the randomization strategy, making sure it is appropriate, and making sure it is implemented correctly.

If you are analyzing an experiment run by others

Whether you should or shouldn't test for a failure of randomization is a matter of opinion. It might depend on your assessment of the likelihood that there is a failure of randomization.

My sense is that post-hoc testing for a failure of randomization might not make a lot of sense, in the absence of some specific reason to suspect a failure of randomization. If you don't trust that the experimenters did the randomization correctly, why do you trust that they did anything else correctly?

If you do detect a failure of randomization, there's not a great answer. If you know that randomization wasn't done properly, then you don't have a randomized controlled trial. You can try to control for the variables that show a difference between the two groups, but at that point you're basically turning it into an observational study, and who is to say that you have controlled for all possible confounders? If a failure of randomization has caused some systematic differences between the treatment vs control groups that you're aware of, there may well be other systematic differences that you're not aware of. So it's a sticky situation.

One other possibility is that perhaps you think randomization was done correctly, but you got unlucky with the split into treatment and control group and an unlikely split caused an imbalance between the two groups: e.g., maybe the treatment got many more older people than the control group, just by bad luck. That could happen. If the sample size is large, the chances that there is a large imbalance is very small, so we could question whether it is worth worrying about it. This is perhaps the one case where you might consider controlling for that factor (age). However, we also know that the probability of this happening by pure chance is very low, so it is questionable whether it is worth the energy to worry about such very-low-probability events. There are many other very-low-probability events that you probably don't worry about, so it's not clear why we'd single this one out.