# Can I use a permutation test to test the null hypothesis ''The difference between two groups is X''?

From what I read on permutation test, the null hypothesis is usually that there is no difference between the two groups.

I want to test if the difference between the mean of the two groups is $$\theta$$ ($$H_0 : \mu_2 - \mu_1 = \theta$$). Can I use a permutation test for that? How could I do it?

I figured that if my null is ''Group A average is 4 units higher than group B average'', maybe I could substract 4 from all values in Group A and then my null would become that there is no difference between the two groups?

My feeling is that I should use another method (maybe bootstrapping) but I am really not sure.

Small precision : my data does not follow normal distribution so that's why I am looking at resampling techniques.

• What is the null hypothesis?
– Tim
Sep 4, 2021 at 14:01
• It is that there is a difference between the two group, and that difference is X. If have many tests to do with similar null hypothesis. So for example, it could be ''Group A average is 4 units higher than group B average'' . Not even sure if it makes sense to state the null as that? Sep 4, 2021 at 15:03
• How about getting a bootstrap CI for the difference and seeing if it includes $2.$ // How much data? // Counts or continuous measurement scale? Sep 4, 2021 at 22:28
• Yeah, bootstrapping might be best. My senior wants us to use permutation tests so that's why I am trying to see if it is possible or not. Depending on the test, we have about 100 to 300 000 rows of data (I have a lot of tests to automate). And the measurements are continuous :) Sep 5, 2021 at 1:12
• Conventionally, uppercase Roman letters represent random variables while Greek letters represent population parameters; I will stick with this convention and use $θ$ where you have $X$. "The difference is $\theta\,$" is too vague, since there are many ways two distributions could differ. If you specify that you intend that the difference in mean is $\theta$; i.e. $H_0:\, \mu_2-\mu_1=\theta$, say, or alternatively specify that the difference in 75th percentiles is $\theta$, or that the median pairwise difference is $\theta$, etc, then you can get somewhere (with some additional assumptions). Sep 5, 2021 at 3:49

With the null hypothesis being that the means differ by some fixed constant, and against a two sided alternative.

There's a couple if ways to approach this. Here's one (which is indeed as you suggest, subtracting the hypothesized difference from the first sample and testing for no difference in means):

Let $$y^*=y-\theta_0$$

Compute $$T = |\bar{y}^* - \bar{x}|$$

Assuming sample sizes are not small, we sample the permutation distribution of the difference in means by randomly drawing new samples without replacement from the combined sample $$(y^*, x)$$ and computing $$T^*_t$$ each time, for $$t=1, 2, ..., B$$, for large $$B$$ - I suggest at least 9999. I typically do many more, since it's generally fast to do.

We then compute the proportion of such statistics (including the original) that equal or exceed $$T$$; this is the p value, approximately.

If the sample sizes are small we can instead compute all possible resamples and get the exact permutation distribution rather than sampling it.

(It is possible to shortcut this a little, allowing us to count only the cases in the tail, but details are beyond an answer of a few paragraphs).