I am trying to produce confidence intervals for a randomization test of no effect and I've found two different methods in the literature. I have a hard time comparing the two and selecting the best one.

Which procedure for creating confidence intervals should be used? What are their respective pros and cons?

Read below for a description of the two methods and references.

The randomization test

The family of sharp null hypotheses:

$H_{0, \tau}$, null hypothesis of effect of size $\tau$: $Y_i(1)=Y_i(0)+\tau$ for all observation $i$, where $Y$ is the possible outcome function.

Typically I want to test the sharp null hypothesis of no effect $H_{0,\tau=0}$ and compute a 95% confidence interval for $\tau$.

The procedure for the hypothesis test of $H_{0,\tau}$:

  • Assume that observations $i\in [1, n]$ were in the control group of the actual experiment, and observations $i\in [n+1, 2n]$ in its treatment group.
  • The observed statistic is: $$ T_{obs} = \frac{1}{n}\sum_{i=n+1}^{2n}Y_i(1)-\frac{1}{n}\sum_{i=1}^{n}Y_i(0) $$
  • The observed statistic is compared to statistics under different assignments. Denote by $\sigma$ a given permutation of $[1, 2n]$: $$ T_{\sigma} = \frac{1}{n}\sum_{i\in [1, 2n],\\\sigma(i)\in [n+1, 2n]}Y_i(1)-\frac{1}{n}\sum_{i\in [1, 2n],\\\sigma(i)\in [1, n]}Y_i(0) $$ Under the null hypothesis, this is computable since $Y_i(1)=Y_i(0)+\tau$ for all $i$.

CI method #1: bring back everything to the "no-effect" case

See this cross-validated question and in this paper.

The procedure (grid version):

  • define a grid $G$ of effect sizes to be tested
  • for every $\tau'\in G$, define the new possible outcome function $\tilde{Y}$: $$ \tilde{Y}_i(0)=\tilde{Y}_i(1)=Y_i(0) \,\,\text{if $i$ was in the control group, $i\in [1, n]$}\\ \tilde{Y}_i(0)=\tilde{Y}_i(1)=Y_i(1)-\tau' \,\,\text{if $i$ was in the treatment group, $i\in [n+1, 2n]$} $$
  • compute the p-value of the zero effect null hypothesis $H_{0,\tau=0}$ on the new possible outcome function $\tilde{Y}$
  • declare that the effect size grid point $\tau'$ is in the confidence interval if the hypothesis test just above is not rejected.

CI method #2: test every effect size

See these course notes.

The procedure (grid version):

  • define a grid $G$ of effect sizes to be tested
  • for every $\tau'\in G$, compute the p-value of the null hypothesis $H_{0,\tau=\tau'}$ that the effect size is equal to $\tau'$
  • declare that the effect size grid point $\tau'$ is in the confidence interval if the hypothesis test just above is not rejected.
  • $\begingroup$ About the first method, there is something that you need very careful about. You are basically working under the Rubin model of potential outcomes, one of the standards to analyze random experiments. However, the hypothesis of interest $H_0: Y_i(1)=Y_i(0)+\tau$ for some $\tau$ is fundamentally untestable since we never get to observe $Y_i(1)$ and $Y_i(0)$ for the same unit $i$ (this is the so-called fundamental problem of causal inference). As a result, we need to modify the aforementioned hypothesis for something like $H_0:F_1(y+\tau)=F_0(y)$ and use a Kolmogorov-Smirnov test statistic. $\endgroup$ Commented Nov 19, 2018 at 20:19
  • $\begingroup$ Thanks for your comment @MauOlivares. Randomization tests as presented here are usually concerned with hypotheses of this type, though they are usually too restrictive. Indeed the null hypothesis $H_0: \forall i Y_i(0)=Y_i(1)$ is way more restrictive than another null hypothesis about distributions or parameters of distributions. Yet randomization tests work quite well in practice and I'd like a way to compute CIs for this type of tests. $\endgroup$
    – jrjd
    Commented Nov 20, 2018 at 12:37
  • $\begingroup$ Beside that, I don't see why your comment concerns the first method more than the other, or than the whole concept of randomization test, who all rely on this type of sharp null hypotheses? $\endgroup$
    – jrjd
    Commented Nov 20, 2018 at 12:41

1 Answer 1


The 2 procedures described are equivalent, but they are indeed coming from 2 different ideas and philosophies.

Let's show that they are indeed similar. We can call each method $m1$ and $m2$.

Let's consider an effect $\tau$.

Let's consider all the potential label permutations $\sigma$ which is a finite set of size $K$.

For a given permutation $\sigma$, let's define a few more variables:

  • $C$ and $T$ which correspond to the set of point that were initially, respectively, in control and treatment.

  • $C_{\sigma}$ and $T_{\sigma}$ which correspond to the set of point that are, under this permutation, in control and treatment.

  • $y_i$ being the initial value of point i. This value can either be from control of from the treatment. $y_i = Y_i(0)$ if $i \epsilon C_i$ and $y_i = Y_i(1)$ if $i \epsilon T_i$

we can compute the associated statistics $$T_{\sigma} = \frac{1}{n} \sum_{i\epsilon T_{\sigma}} Y_i(1) - \frac{1}{n} \sum_{i\epsilon C_{\sigma}} Y_i(0) $$

$$T_{\sigma} = \frac{1}{n} ( \sum_{i\epsilon T_{\sigma}\\i\epsilon C} Y_i(1) + \sum_{i\epsilon T_{\sigma}\\i\epsilon T} Y_i(1) - \sum_{i\epsilon C_{\sigma}\\i\epsilon C} Y_i(0) + \sum_{i\epsilon C_{\sigma}\\i\epsilon T} Y_i(0) )$$

We can now compute that statistics in both case $m1$ and $m2$ and considering an effect $\tau$.

$$T_{\sigma}^{m1} = \frac{1}{n} ( \sum_{i\epsilon T_{\sigma}\\i\epsilon C} y_i + \sum_{i\epsilon T_{\sigma}\\i\epsilon T} (y_i - \tau) - \sum_{i\epsilon C_{\sigma}\\i\epsilon C} y_i - \sum_{i\epsilon C_{\sigma}\\i\epsilon T} (y_i-\tau) )$$

$$T_{\sigma}^{m2} = \frac{1}{n} ( \sum_{i\epsilon T_{\sigma}\\i\epsilon C} (y_i+\tau) + \sum_{i\epsilon T_{\sigma}\\i\epsilon T} y_i - \sum_{i\epsilon C_{\sigma}\\i\epsilon C} y_i - \sum_{i\epsilon C_{\sigma}\\i\epsilon T} (y_i-\tau) )$$

$$T_{\sigma}^{m2} - T_{\sigma}^{m1} = \frac{1}{n} (\sum_{i\epsilon T_{\sigma}\\i\epsilon C} \tau + \sum_{i\epsilon T_{\sigma}\\i\epsilon T} \tau) $$

$$T_{\sigma}^{m2} - T_{\sigma}^{m1} = \frac{1}{n} * \tau \sum_{i\epsilon T_{\sigma}} 1 $$

So the 2 permutations distribution are shifted by $T_{\sigma}^{m2} - T_{\sigma}^{m1}$

The final step is to show that even though the permutation distributions are shifted, the observed value are also shifted, making the p-value and CI interval the same.

$$T_{obs}^{m2} = \frac{1}{n} * ( \sum_{i\epsilon T_{\sigma}} y_i - \sum_{i\epsilon C_{\sigma}} y_i) $$

$$ T_{obs}^{m1} = \frac{1}{n} * ( \sum_{i\epsilon T_{\sigma}} (y_i-\tau) - \sum_{i\epsilon C_{\sigma}} y_i) $$

$$T_{obs}^{m2} - T_{obs}^{m1} = \frac{1}{n} * \tau \sum_{i\epsilon T_{\sigma}} 1 $$

This for each effect, $\tau$ both process will land the exact same p-value because the permutation distribution and the observed statistic are shifted by the same amount.

  • $\begingroup$ Thanks Romain! I would have had a hard time figuring out that the two procedures were equivalent! $\endgroup$
    – jrjd
    Commented Nov 22, 2018 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.