Hypothesis Testing: Permutation Testing Justification

Context: Hypothesis test using permutation testing about 2 populations, assuming I sample $k_{1}$ and $k_{2}$ items from population 1 and 2, respectively, where $k_{1}, k_{2} \in \mathbb{Z}^{\geq 1}$:

$H_{o}$: $m_{1} = m_{2}$; $H_{a} : m_{1} \not = m_{2}$ where $m_{x}$ refers to the median of population $x$.

Concern: In permutation testing, we'd compute the test statistic of interest ($\bar{m_{1}} - \bar{m_{2}}$) for each possible permutation (i.e. $(k_{1} + k_{2})!$) and use the histogram from these values as the null distribution (i.e. sampling distribution of the mentioned test statistic under the null hypothesis), using it to come up with a p-value. This seems to imply that by assuming the null hypothesis above, each permutation of our sample is equally likely, and it is this with which I do not agree. It seems to me to make this conclusion, the null hypothesis would have to be much more general (i.e. that the two populations have identical and independent distributions, as mentioned in http://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704_nonparametric/bs704_nonparametric2.html), but then this would also mean that rejecting the null hypothesis would be rejecting of a much broader hypothesis and not our desired hypothesis that the medians are the same (i.e. the two populations don't have the identical and independent distributions, which may or may not mean that they have the same median).

Summary of Concern: What justifies using the distribution of the test statistic from permuting our samples to represent the null distribution above?

I am assuming that there is some justification because I think we carried out tests in this manner in my stats course, but I never paused to reflect on the steps of the procedure to make sure I fully understood them.

Also a side question: When the observed test statistic is 0, do you reject the null hypothesis or fail to reject the null hypothesis? (I've read that always assume that data is consistent with the null hypothesis by default, unless your data has an observed test statistic value that aligns with the alternative hypothesis and that is statistically significant.)

2 Answers

Generally hypothesis tests are accompanied by extra assumptions that will need to hold (at least when the null is true), so that the null distribution of the test statistic can be obtained; this is as true for nonparametric tests as for parametric ones.

So for example, the usual two sample t-test comes with assumptions of equality of variance and independence - which we rely on when finding the null distribution of the test statistic - even though neither condition is in the hypothesis itself.

For permutation tests you need add at least the assumptions required to obtain exchangeability, though typically we'd use somewhat stronger assumptions (like independence for example).

• To summarize: We make additional "hidden" assumptions about our sampling procedure (e.g. random sample w/ replacement) and populations (e.g. independent and equally distributed) to narrow down what the null distribution could look like. This seems fair given that after all we are trying to reason under uncertainty. But this makes me wonder: am I correct in saying that the more these hidden assumptions stray from reality of the population and sampling procedure, the more prone we are to making type I and type II errors? – Coder47 Feb 11 '17 at 16:27
• Also, if possible, could you please answer the side question, too? – Coder47 Feb 11 '17 at 16:29
• Another concern: I can't think of a specific example of a set of reasonable assumptions to make about the sampling procedure and populations that would ensure the sampling distribution of the test statistic (will refer to this distribution as p-dist from now on) would be roughly identical to the true null distribution, so as to avoid errors in conclusions. In the case that sample sizes are essentially the size of the original populations, then I am convinced that the p-dist will be roughly equal to the null distribution, but in the case that the sample size is small, I am not. – Coder47 Feb 11 '17 at 23:24
• Let us assume that randomly sampling w/ replacement from two independent and equally distributed populations. Under this assumption, if I can somehow show that this implies that most samples will have a p-dist that is similar to our original sample's p-dist, then I will be convinced that the true null distribution is roughly the same as our original p-dist, since the null distribution is an average of the p-dist of all possible samples. Unfortunately, I cannot prove this implication to myself, and I feel it is strange to just assume the resultant of the implication. – Coder47 Feb 11 '17 at 23:30

You are right that you are testing much broader assumption that the group labels are "random" and play no role in your results. As a proxy to test such hypothesis you use some test statistic that is evaluated on the permuted samples. In this case it is median, but it could something else as well. So you test if the labels are exchangeable as tested using median as criterion to assess that. In the end you will learn how "likely" would it be to find different medians in both groups if the group labels played no role in your data.