What kind of null hypothesis can a permutation test consider?

Question

From what I have seen, permutation tests are always used to compare the means (or median) of two different groups in the experiment. Even Wikipedia says

A permutation test involves two or more samples. The null hypothesis is that all samples come from the same distribution.

Is it possible to extend permutation test for just a single sample to estimate a population parameter? If not, why?

Answer 1

Permutation tests can be done (contrary to the implication of the quoted statement [1]) for single samples and intervals can be constructed from them. Indeed, you may well have done one without realizing it. Further, permutation tests can be performed for a very large array of situations beyond those connected to means or medians [2].

1. You can use permutation tests to construct intervals for parameters. The set of hypothesized population parameter values that would not be rejected by the test at a level not exceeding $\alpha$ will form an interval with coverage at least $1-\alpha$ for the parameter, under the same assumptions as needed for the test.

   For example, a two-sample permutation test of means can be used to form an interval for the difference in means. A Wilcoxon-Mann-Whitney two sample test can be used to obtain an interval for the population median of pairwise cross-population differences, and so on.

2. One sample permutation tests certainly exist. Intervals for corresponding population quantities can be constructed from them.

   For example you could do a permutation test for $H_0: \mu=\mu_0$ against the usual two-sided inequality alternative. This requires some assumptions under $H_0$; the usual test constructs exchangeable quantities by assuming symmetry about the assumed location parameter $\mu_0$, so that $X-\mu_0$ and $\mu_0-X$ may be substituted for each other (i.e. the sign of $X-\mu_0$ may be flipped) without changing the distribution when $H_0$ is true. This is not the only possible such assumption and corresponding exchangeable quantity.

   We can then consider the set of possible $\mu_0$ values that would not lead to rejection of $H_0$ using this test, at some level $\alpha$. This non-rejection region (which some authors have called a consonance region) may be used as an interval for the parameter.

   Similarly, intervals may be constructed for many other parameters.

3. At least one such one-sample permutation-test interval is built into many statistics packages; that from a Wilcoxon signed rank test (almost all rank-based tests are simply permutation tests based on rank-transformed data). Here's an example of it in R.

     x = round(rlogis(10,15,3),2)  # create some data
     x
     [1]  6.87  8.93 18.73  9.41 11.88 15.11 26.92  6.81 18.46 15.41
   
     wilcox.test(x, conf.int=TRUE)
   
            Wilcoxon signed rank exact test
   
    data:  x
    V = 55, p-value = 0.001953
    alternative hypothesis: true location is not equal to 0
    95 percent confidence interval:
      9.17 18.46
    sample estimates:
    (pseudo)median 
            13.495 
   

   Here we see that the estimate of the population pseudomedian is 13.495; the 95% interval for the population pseudomedian is (9.17, 18.46). In this case we know the population pseudomedian because we generated the data -- it was 15. The interval happened to overlap the population value in this instance.

---

[1]: Beware taking everything about statistics on Wikipedia as gospel; while many of its stats articles are pretty good (for example the articles on individual distributions are often quite good), not everyone who edits Wikipedia knows what they're talking about and many of the stats articles (particularly the ones that cover less technical material) are frequently edited by people whose desire to help exceeds their grasp of the subject. Worse, there's a lot of books and articles about statistics that contain many serious errors so it's also easy for those helpful people to provide references (albeit wrong or misleading ones) for the claims in their edits.

[2]: However, I would add that often it's a good idea to construct a scaled (e.g. studentized) version of the statistic or similarly scaled/normalised quantity rather than use a raw statistic, such as using a t-statistic rather than a raw difference in means; the properties of the test are sometimes improved. One example would be with a test for the Pearson correlation