How can we interpret differences between t test p values and randomization-inference p values?
Let’s say we have a randomized experiment with a binary treatment, denoted $Z_i = 1$ if unit $i$ is assigned to treatment, and outcomes, denoted $Y_i$.
We want test for a treatment effect.
We test both the sharp null hypothesis of no effect and the null hypothesis of no average effect.
Definition $H_{0,sharp}$: Sharp null hypothesis of no effect The treatment effect is zero for all subjects. Formally, $Y_i(1) = Y_i(0)$ for all $i$.
Definition $H_{0,weak}$: Null hypothesis of no average effect (sometimes called the weak null hypothesis) The average treatment effect is zero. Formally, $\mu_{Y(1)} = \mu_{Y(0)}$.
We test $H_{0,sharp}$ using randomization inference (RI) and we test $H_{0,weak}$ with a t test.
If we run these two tests and get different answers, what are useful ways to interpret differences between the t test p value and the RI p value?
Strictly speaking, the two procedures test different hypotheses and they cannot be meaningfully compared, but this is not very useful, and will not satisfy non-specialists (people with substantive rather than technical interest in your research) who want to understand why your results look different when using RI or a t test. Furthermore, the two tests are alternative approaches to answer the same substantive question, “was there a treatment effect?” We should have a guidelines for thinking about different answers to the same substantive question.
A good answer would have a general enough discussion of differences to encompass differences in p values that would lead us to different statistical conclusions (e.g., one test p<0.05 and the other p>0.05) and those that would lead to make the same conclusion from both tests (e.g., both tests p<0.05 or both tests p>0.05).
Notes on RI
For those unfamiliar with RI: The RI p value is calculated by, first, computing the distribution of the test statistic across all (or many) treatment assignments, which is called the null or randomization distribution. The RI p value denotes the proportion of the randomization distribution that is larger than our observed test statistic. (More discussion here, particularly page 5.)
We can conduct RI by calculate the test statistic for all possible permuted treatment assignment vectors (to calculate an exact RI p value) or using a large sample of permuted treatment assignment vectors (to calculate an asymptotic RI p value). As Gerber and Green (2012) write, “Whether one uses all possible randomizations or a large sample of them, the calculation of p values based on an inventory of possible randomizations is called randomization inference.