Choose between several confidence interval procedures for randomization test I am trying to produce confidence intervals for a randomization test of no effect and I've found two different methods in the literature.
I have a hard time comparing the two and selecting the best one.
Which procedure for creating confidence intervals should be used? What are their respective pros and cons?
Read below for a description of the two methods and references.
The randomization test
The family of sharp null hypotheses:
$H_{0, \tau}$, null hypothesis of effect of size $\tau$: $Y_i(1)=Y_i(0)+\tau$ for all observation $i$, where $Y$ is the possible outcome function.
Typically I want to test the sharp null hypothesis of no effect $H_{0,\tau=0}$ and compute a 95% confidence interval for $\tau$.
The procedure for the hypothesis test of $H_{0,\tau}$:


*

*Assume that observations $i\in [1, n]$ were in the control group of the actual experiment, and observations $i\in [n+1, 2n]$ in its treatment group.

*The observed statistic is:
$$
T_{obs} = \frac{1}{n}\sum_{i=n+1}^{2n}Y_i(1)-\frac{1}{n}\sum_{i=1}^{n}Y_i(0)
$$

*The observed statistic is compared to statistics under different assignments. Denote by $\sigma$ a given permutation of $[1, 2n]$:
$$
T_{\sigma} = \frac{1}{n}\sum_{i\in [1, 2n],\\\sigma(i)\in [n+1, 2n]}Y_i(1)-\frac{1}{n}\sum_{i\in [1, 2n],\\\sigma(i)\in [1, n]}Y_i(0)
$$
Under the null hypothesis, this is computable since $Y_i(1)=Y_i(0)+\tau$ for all $i$.


CI method #1: bring back everything to the "no-effect" case
See this cross-validated question and in this paper.
The procedure (grid version):


*

*define a grid $G$ of effect sizes to be tested

*for every $\tau'\in G$, define the new possible outcome function $\tilde{Y}$:
$$
\tilde{Y}_i(0)=\tilde{Y}_i(1)=Y_i(0) \,\,\text{if $i$ was in the control group, $i\in [1, n]$}\\
\tilde{Y}_i(0)=\tilde{Y}_i(1)=Y_i(1)-\tau' \,\,\text{if $i$ was in the treatment group, $i\in [n+1, 2n]$}
$$

*compute the p-value of the zero effect null hypothesis $H_{0,\tau=0}$ on the new possible outcome function $\tilde{Y}$

*declare that the effect size grid point $\tau'$ is in the confidence interval if the hypothesis test just above is not rejected.


CI method #2: test every effect size
See these course notes.
The procedure (grid version):


*

*define a grid $G$ of effect sizes to be tested

*for every $\tau'\in G$, compute the p-value of the null hypothesis $H_{0,\tau=\tau'}$ that the effect size is equal to $\tau'$

*declare that the effect size grid point $\tau'$ is in the confidence interval if the hypothesis test just above is not rejected.

 A: The 2 procedures described are equivalent, but they are indeed coming from 2 different ideas and philosophies.
Let's show that they are indeed similar. We can call each method $m1$ and $m2$.
Let's consider an effect $\tau$.
Let's consider all the potential label permutations $\sigma$ which is a finite set of size $K$.
For a given permutation $\sigma$,  let's define a few more variables:


*

*$C$ and $T$ which correspond to the set of point that were initially, respectively, in control and treatment.

*$C_{\sigma}$ and $T_{\sigma}$ which correspond to the set of point that are, under this permutation, in control and treatment.

*$y_i$ being the initial value of point i. This value can either be from control of from the treatment. $y_i = Y_i(0)$ if $i \epsilon C_i$ and $y_i = Y_i(1)$ if $i \epsilon T_i$
we can compute the associated statistics $$T_{\sigma} = \frac{1}{n} \sum_{i\epsilon T_{\sigma}} Y_i(1) - \frac{1}{n} \sum_{i\epsilon C_{\sigma}} Y_i(0) $$
$$T_{\sigma} = \frac{1}{n} ( \sum_{i\epsilon T_{\sigma}\\i\epsilon C} Y_i(1) + \sum_{i\epsilon T_{\sigma}\\i\epsilon T} Y_i(1) -  \sum_{i\epsilon C_{\sigma}\\i\epsilon C} Y_i(0) + \sum_{i\epsilon C_{\sigma}\\i\epsilon T} Y_i(0) )$$
We can now compute that statistics in both case $m1$ and $m2$ and considering an effect $\tau$.
$$T_{\sigma}^{m1} = \frac{1}{n} ( \sum_{i\epsilon T_{\sigma}\\i\epsilon C} y_i + \sum_{i\epsilon T_{\sigma}\\i\epsilon T} (y_i - \tau) -  \sum_{i\epsilon C_{\sigma}\\i\epsilon C} y_i - \sum_{i\epsilon C_{\sigma}\\i\epsilon T} (y_i-\tau) )$$
$$T_{\sigma}^{m2} = \frac{1}{n} ( \sum_{i\epsilon T_{\sigma}\\i\epsilon C} (y_i+\tau) + \sum_{i\epsilon T_{\sigma}\\i\epsilon T} y_i -  \sum_{i\epsilon C_{\sigma}\\i\epsilon C} y_i - \sum_{i\epsilon C_{\sigma}\\i\epsilon T} (y_i-\tau) )$$
$$T_{\sigma}^{m2} - T_{\sigma}^{m1} =  \frac{1}{n} (\sum_{i\epsilon T_{\sigma}\\i\epsilon C} \tau + \sum_{i\epsilon T_{\sigma}\\i\epsilon T} \tau) $$
$$T_{\sigma}^{m2} - T_{\sigma}^{m1} =  \frac{1}{n} * \tau \sum_{i\epsilon T_{\sigma}} 1  $$
So the 2 permutations distribution are shifted by $T_{\sigma}^{m2} 
 - T_{\sigma}^{m1}$
The final step is to show that even though the permutation distributions are shifted, the observed value are also shifted, making the p-value and CI interval the same.
$$T_{obs}^{m2} =  \frac{1}{n} * ( \sum_{i\epsilon T_{\sigma}} y_i - \sum_{i\epsilon C_{\sigma}} y_i) $$
$$ T_{obs}^{m1} =  \frac{1}{n} * ( \sum_{i\epsilon T_{\sigma}} (y_i-\tau) - \sum_{i\epsilon C_{\sigma}} y_i) $$
$$T_{obs}^{m2} - T_{obs}^{m1} =  \frac{1}{n} * \tau \sum_{i\epsilon T_{\sigma}} 1  $$
This for each effect, $\tau$ both process will land the exact same p-value because the permutation distribution and the observed statistic are shifted by the same amount.
