t test p value vs randomization-inference p value: What can we learn from comparison?

How can we interpret differences between t test p values and randomization-inference p values?

Let’s say we have a randomized experiment with a binary treatment, denoted $Z_i = 1$ if unit $i$ is assigned to treatment, and outcomes, denoted $Y_i$.

We want test for a treatment effect.

We test both the sharp null hypothesis of no effect and the null hypothesis of no average effect.

Definition $H_{0,sharp}$: Sharp null hypothesis of no effect The treatment effect is zero for all subjects. Formally, $Y_i(1) = Y_i(0)$ for all $i$.

Definition $H_{0,weak}$: Null hypothesis of no average effect (sometimes called the weak null hypothesis) The average treatment effect is zero. Formally, $\mu_{Y(1)} = \mu_{Y(0)}$.

We test $H_{0,sharp}$ using randomization inference (RI) and we test $H_{0,weak}$ with a t test.

If we run these two tests and get different answers, what are useful ways to interpret differences between the t test p value and the RI p value?

Strictly speaking, the two procedures test different hypotheses and they cannot be meaningfully compared, but this is not very useful, and will not satisfy non-specialists (people with substantive rather than technical interest in your research) who want to understand why your results look different when using RI or a t test. Furthermore, the two tests are alternative approaches to answer the same substantive question, “was there a treatment effect?” We should have a guidelines for thinking about different answers to the same substantive question.

A good answer would have a general enough discussion of differences to encompass differences in p values that would lead us to different statistical conclusions (e.g., one test p<0.05 and the other p>0.05) and those that would lead to make the same conclusion from both tests (e.g., both tests p<0.05 or both tests p>0.05).

Notes on RI

For those unfamiliar with RI: The RI p value is calculated by, first, computing the distribution of the test statistic across all (or many) treatment assignments, which is called the null or randomization distribution. The RI p value denotes the proportion of the randomization distribution that is larger than our observed test statistic. (More discussion here, particularly page 5.)

We can conduct RI by calculate the test statistic for all possible permuted treatment assignment vectors (to calculate an exact RI p value) or using a large sample of permuted treatment assignment vectors (to calculate an asymptotic RI p value). As Gerber and Green (2012) write, “Whether one uses all possible randomizations or a large sample of them, the calculation of p values based on an inventory of possible randomizations is called randomization inference.

• Not quite sure what "conventional p-value" means here. I think there are two distinctions to be made: (1) between the average treatment effect's being zero & the effect for each unit's being zero, & (2) between inference based on sampling & inference based on random assignment of treatments. – Scortchi - Reinstate Monica Jun 26 '17 at 16:35
• @Scortchi I'll try to make the question clearer. First, I improved my definition of terms and I changed the adjective from conventional to asymptotic. Another adjectives could be parametric. Is this clear now what p values I'm contrasting to randomization-inference-based p values? – Dr. Beeblebrox Jun 26 '17 at 17:35
• @Dr.Beeblebrox I'd like to edit my answer, but it would be helpful to define some terms. Can you write the so-called sharp null hypothesis out and define the notation? Can you also describe the "randomization inference p-value"? Do you consider Fisher's Exact Test to be a special case of such a p-value? – AdamO Jun 26 '17 at 18:47
• @AdamO I added some notation and defined RI p-value. I didn't add discussion of the relationship to Fisher's Exact Test because of length. On that point, I understand randomization and permutation inference to be modern procedures derived from Fisher's exact test. Perm. Inf. for when you can calculate test statistics for all treatment assignment permutations (that is, for a binary treatment with n subjects, $2^n$ is computationally workable). RI is for when you sample draws from a very large set of $2^n$ treatment asgnmnt vectors. Discussion here stats.stackexchange.com/q/55742/22088 – Dr. Beeblebrox Jun 26 '17 at 19:16
• In general, different answers from different tests can just arise from minute differences. Remember that any test statistic and p-value is just a random variable that will invariably vary even across absolutely identical experiments. So, jumping from just below 0.05 to just over 0.05 for a p-value is a completely meaningless difference, even if some people will wrongly interpret a lot into it. – Björn Apr 18 '18 at 15:28

I think your statement of the randomization inference null hypothesis is incorrect. Or at least, you're confusing two methods to test hypotheses versus two different hypotheses. The randomization test aka the permutation test considers the exact or approximation distribution of test statistics obtained when "labels" are randomly swapped between treatment/control subjects. This can be used to test the weak null hypothesis of no average treatment effect by calculating the t-test statistic for each permuted dataset and evaluating the proportion of these exceeding the one obtained in the unpermuted dataset.

In this working article they frame the hypothesis of treatment effect variation as one of homogeneity where the average treatment effect is considered a nuisance parameter: basically "I don't care whether this drug works, I just want to know if it works differently in some people than it does in others." The effect for the first hypothesis, tested using usually analysis of parallel design, is called average treatment effect (ATE) and the second hypothesis here has been called treatment effect variation (TEV). Testing for TEV smells of a test of effect modification, in the absence of a known effect modifier, and resembles subgroup analysis. Using randomization tests for TEV is a novel and interesting method to consider and is worth reading this article in depth to understand how exactly they formulated such a test.

To summarize how the two hypothesis might agree or disagree in a $2 \times 2$ table:

Case 1: ATE no TEV: the drug works and it has the same potential outcome in everybody regardless. Solution: do not recommend if harmful, consider effect size before recommending approval/use.

Case 2: no ATE no TEV: the drug does not work in anyone. Solution: conclude drug is futile relative to standard of care.

Case 3: no ATE, TEV: the drug works in individuals in such a contrived way that the harm in some and the benefit in others is completely balanced. Solution: identify indicators/contraindicators of harm/benefit subgroups and conduct follow-up study if predicted benefit is of clinical significance.

Case 4: ATE, TEV: the drug shows some average effect but this effect is not the same in everyone. Solution: identify harm groups if any and establish contraindications, predict benefit in remaining group and conduct follow-up study if it is of clinical significance.

• My definition of the RI p value is just derived from a different application of RI than the one you describe. Generally, RI is used to test the sharp null hypothesis of constant treatment effects for all units (which can be used to study variation per the paper you linked); RI can also be used to test the sharp null of no effect for all units, which is the application I am talking about. See here, page 2, mattblackwell.org/files/teaching/s05-fisher.pdf – Dr. Beeblebrox Jun 26 '17 at 18:15
• I edited my question to be more precise – Dr. Beeblebrox Jun 28 '17 at 21:58

I found this discussion of the difference between t-test p values and RI p values helpful, and it speaks to the question I ask above.

Author: Don Green Source: https://egap.org/methods-guides/10-things-randomization-inference

Randomization inference may give different p-values from conventional tests when the number of observations is small and when the distribution of outcomes is non-normal

Conventional p-values typically rely on approximations that presuppose either that the outcomes are normally distributed or that the subject pool is large enough that the test statistics follow a posited sampling distribution. When outcomes are highly skewed, as in the case of donations (a few people donate large sums, but the overwhelming majority donate nothing), conventional methods may produce inaccurate p-values. Gerber and Green, Field Experiments, (2012, p.65) give the following example in which randomization inference and conventional test statistics produce different results: