2
$\begingroup$

From what I read on permutation test, the null hypothesis is usually that there is no difference between the two groups.

I want to test if the difference between the mean of the two groups is $\theta$ ($H_0 : \mu_2 - \mu_1 = \theta$). Can I use a permutation test for that? How could I do it?

I figured that if my null is ''Group A average is 4 units higher than group B average'', maybe I could substract 4 from all values in Group A and then my null would become that there is no difference between the two groups?

My feeling is that I should use another method (maybe bootstrapping) but I am really not sure.

Small precision : my data does not follow normal distribution so that's why I am looking at resampling techniques.

Improve this question
$\endgroup$
6
  • $\begingroup$ What is the null hypothesis? $\endgroup$
    – Tim
    Commented Sep 4, 2021 at 14:01
  • $\begingroup$ It is that there is a difference between the two group, and that difference is X. If have many tests to do with similar null hypothesis. So for example, it could be ''Group A average is 4 units higher than group B average'' . Not even sure if it makes sense to state the null as that? $\endgroup$
    – Capucine25
    Commented Sep 4, 2021 at 15:03
  • $\begingroup$ How about getting a bootstrap CI for the difference and seeing if it includes $2.$ // How much data? // Counts or continuous measurement scale? $\endgroup$
    – BruceET
    Commented Sep 4, 2021 at 22:28
  • $\begingroup$ Yeah, bootstrapping might be best. My senior wants us to use permutation tests so that's why I am trying to see if it is possible or not. Depending on the test, we have about 100 to 300 000 rows of data (I have a lot of tests to automate). And the measurements are continuous :) $\endgroup$
    – Capucine25
    Commented Sep 5, 2021 at 1:12
  • 1
    $\begingroup$ Conventionally, uppercase Roman letters represent random variables while Greek letters represent population parameters; I will stick with this convention and use $θ$ where you have $X$. "The difference is $\theta\,$" is too vague, since there are many ways two distributions could differ. If you specify that you intend that the difference in mean is $\theta$; i.e. $H_0:\, \mu_2-\mu_1=\theta$, say, or alternatively specify that the difference in 75th percentiles is $\theta$, or that the median pairwise difference is $\theta$, etc, then you can get somewhere (with some additional assumptions). $\endgroup$
    – Glen_b
    Commented Sep 5, 2021 at 3:49

1 Answer 1

Reset to default
1
$\begingroup$

With the null hypothesis being that the means differ by some fixed constant, and against a two sided alternative.

There's a couple if ways to approach this. Here's one (which is indeed as you suggest, subtracting the hypothesized difference from the first sample and testing for no difference in means):

Let $y^*=y-\theta_0$

Compute $T = |\bar{y}^* - \bar{x}|$

Assuming sample sizes are not small, we sample the permutation distribution of the difference in means by randomly drawing new samples without replacement from the combined sample $(y^*, x)$ and computing $T^*_t$ each time, for $t=1, 2, ..., B$, for large $B$ - I suggest at least 9999. I typically do many more, since it's generally fast to do.

We then compute the proportion of such statistics (including the original) that equal or exceed $T$; this is the p value, approximately.

If the sample sizes are small we can instead compute all possible resamples and get the exact permutation distribution rather than sampling it.

(It is possible to shortcut this a little, allowing us to count only the cases in the tail, but details are beyond an answer of a few paragraphs).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.