Advantages of using t-value as test statistic in permutation tests?

Question

I'm working with some permutation tests where my main aim is to evaluate a treatment effect, and I have a question about choice of test statistic.

I've seen that some use β for the variable of interest as the test statistic, while others use the t-value (β/SE β) as test statistic.

I've not seen good explanations of advantages of choosing one over the other, and wonder if someone can tip me on relevant references or provide a quick explanation.

The best I've seen so far is here (p. 6):

Suppose that we really are interested in the standardized difference between means, but we are reluctant to use the parametric t-test because it assumes that the sampling distribution for t values is based on an underlying normal distribution. With resampling we can use a calculated t-value as a measure of the group difference, but we can test it against an empirical sampling distribution for the t-value ... We can randomly reshuffle the data into two groups of N=20 each and recompute the t-value. If we do this for many reshuffles of the data (e.g., 10,000) we can generate an empirical distribution of the t-value. This distribution is NOT necessarily distributed according to the parametric t distribution

From what I gather, the main point is that by using the t-value as a test statistic the assumption of an underlying normal distribution can be relaxed. If so, are there any other arguments for or against using the t-value?

I see that the t-value is also used in permutation tests in Efron & Hastie (2017) Computer Age Statistical Inference, p. 49-50.

Answer 1

The main reason to use a t-value (and, indeed, any approximately pivotal test statistic) in a permutation test is to give the test asymptotic validity in the case of unequal variances. Given that you always want this property, you should always base permutation tests on β/(SE β) rather than just β alone.

This property was first described in Janssen, 1997. As many textbooks and papers note, the ordinary permutation test is only "exact" for the test of identical distributions. Typically, however, we want to test for equality of the parameter of interest, not that the distributions are identical. More importantly, we also generally want to make directional conclusions about the results of the test. Janssen (and later, Chung and Romano) pointed out that in order to do this you have to use a pivotal test statistic (which is related to why the bootstrap-t functions better than the ordinary bootstrap).

In order to make an approximately pivotal test statistic, you can divide a comparison of interest with an estimate of its standard error (called Studentization). The t-value is the classic example of this procedure. Given that the null hypothesis of "equal distributions" is rarely interesting, you should ALWAYS be using an approximately pivotal test statistic. Note, however, that is sometimes difficult to estimate the standard error of a comparison (though you can always nest a bootstrap inside of your permutation test).