3
$\begingroup$

The observational case

For observational data, Hernán & Robins (2023, p. 49) state:

In the absence of marginal randomization, [computing the average causal effect in the entire population] requires adjustment for the variables $L$ that ensure conditional exchangeability of the treated and untreated

Adjustment can then be achieved via regression adjustment in an outcome model, standardization, stratification/restriction, matching or IP weighting. When using matching or weighting, for example, packages offer a multitude of methods and diagnostics to evaluate the degree of balance achieved.

If our generated pseudo-population is greatly imbalanced, we will consider our estimate most surely biased and, thus, invalid.

The experimental case

In the context of randomized clinical trials, Senn states explicitly in myth number 2 of the article "Seven myths of randomisation in clinical trials" (p. 1441):

There is no requirement for baseline balance for valid inference.

Likewise, in his guest blog post "Randomisation is not about balance, nor about homogeneity but about randomness", he unambiguously states:

  1. Randomisation does not produce balance
  2. This does not affect the validity of the analysis

, as well as,

Balance is a matter of precision not validity


The question

How does the process by which data is created alters the role of balance, from one of ensuring validity into one of ensuring precision?

$\endgroup$

1 Answer 1

3
$\begingroup$

In short

Balance is necessary but not sufficient for observational data, as most of the "work" is done by the causal assumptions. Whereas in experimental studies, balance is not as irrelevant as suggested by the quotes from Senn, as observed covariates have to be taken into account even if there's randomization.


The confusion in the question usually has the implicit erroneous reasoning that in observational data:

  1. Identification of the causal effect requires adjusting for some variables

  2. Adjustment on $L$ will usually involve some balancing of units across levels of $L$

  3. Thus, the balancing itself is identifying the causal effect by inducing conditional exchangeability

Balancing is a statistical property, and it cannot on its own identify a causal effect. We will always need to appeal to causal assumptions.

The issue, as common to most attempts of drawing causal inferences, is unobserved confounding.

The observational case

Note that the observational case assumes already that $L$ ensures conditional exchangeability, i.e. that $L$ is a sufficient adjustment set (and thus, that there is no unmeasured confounding after adjusting for $L$). The very same property of "sufficiency" is what we are assuming. Without this causal assumption, balance does not achieve valid inferences of a causal effect. Balancing is then just a way to assess how well we are adjusting for $L$.

We introduced exchangeability as an assumption.

The experimental case

Here, the design introduced exchangeability into our data.

Remember that exchangeability is a property of the distribution of (potential) outcomes—not the covariates. Thus, as Senn states in the comments in the blog post:

RCTs are good at dealing with confounding, not because confounders are eliminated but because their joint effect can be estimated.

Thus, when randomizing, we do not care about adjusting for confounders for validity, nor we worry about unobserved confounders—there cannot be confounders (as in "common causes of exposure and outcome") since randomization is not caused by any covariate by definition!

We can still have prognostic factors, i.e. causes of the outcome, that can be unbalanced across treatment arms. And note that myth 6 of the article clearly states that one must not ignore them just because we randomized (p. 1446):

In fact, it is not logical to ignore prognostic covariates even if they are perfectly balanced once they have been observed. Valid inference depends on conditioning on what is known. It is only when the actual distribution of covariates has not been observed that it is acceptable to substitute their distribution in probability. [...] Obviously, what has not been measured cannot be put in the model, and it is here that randomisation proves valuable. However, if it was chosen to measure something because it was prognostic, then it is not reasonable to behave as if one had not seen it.

We would thus include them in the model or perform post-stratification, so balance is not irrelevant in the experimental case.


However

Consider the critic that Greenland, Robins & Pearl (1999, p. 35, bolding mine) raise regarding randomization:

Neither restriction nor matching prevents (although they may diminish) imbalances on unrestricted, unmatched or unmeasured covariates. In contrast, randomized treatment allocation (randomization) offers a means of dealing with confounding by covariates not explicitly accounted for by the design. It must be emphasized, however, that this solution is only probabilistic and subject to severe practical constraints. [...] Thus, even in a perfectly executed randomized trial, the no-confounding condition $\mu_{A0} = \mu_{B0}$ is not a realistic assumption for inferences about causal effects. Succesful randomization simply insures (sic) that the difference $\mu_{A0} - \mu_{B0}$ and hence the degree of confounding, has expectation zero and converges to zero under the randomization distribution; it also provides a permutation distribution for causal inferences (Fisher, 1935; Cox, 1958, Chapter 5).

To which Senn would likely reply (p. 1442, bolding mine):

In fact, the probability calculation applied to a clinical trial automatically makes an allowance for the fact that groups will almost certainly be unbalanced, and if one knew that they were balanced, then the calculation that is usually performed would not be correct. Every statistician knows that you should not analyse a matched pairs design as if it were a completely randomised design. In the former, there is deliberate balancing by the device of matching; in the latter, there is not. In other words, not only despite but also because we know that in practice many hidden confounders will be unbalanced, the conventional analysis of randomised trials is valid. If we knew these factors to be balanced, then the conventional analysis would be invalid.

The defense of Senn seems relatively weak, and in the rest of the section in the paper he seems to imply that given that there is no better alternative, we should be content with taking shelter in the randomization distribution. Personally, I feel this is not entirely honest or transparent to the interpretation that readers will give to the results of an RCT, but maybe that is the subject for another question altogether...

$\endgroup$
4
  • 2
    $\begingroup$ It might help to put a bold summary at the top. My first read of this made me think you dispute the claim, yet now I am not sure. $\endgroup$
    – Dave
    Apr 19, 2023 at 23:41
  • $\begingroup$ Hope it is more clear now! $\endgroup$
    – Kuku
    Apr 23, 2023 at 15:34
  • $\begingroup$ very helpful research. I'm confused what's the difference between 「observed covariates have to be taken into account 」vs 「Senn myth 3: Observed covariates can be ignored」? $\endgroup$
    – wei
    Apr 24, 2023 at 1:59
  • $\begingroup$ @wei Thanks! The fact that observed covariates [known to be prognostic factors] have to be taken into account in the model is the conclusion of why "Observed covariates may be ignored because one has randomised" is a myth (the sixth and not the third one). $\endgroup$
    – Kuku
    Aug 24, 2023 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.