Let a vector of i.i.d random varibles $(X_1,X_2,X_3,\cdots, X_m)$ and another vector of i.i.d rvs $(Y_1,Y_2,Y_3,\cdots, Y_n)$ be given. Suppose $X_i$ stands for the recovery time using a new treatment and $Y_i$ be the recovery time with the old treatment. Suppose we are interested in running a hypothesis test with the following null hypothesis $\mathbb{E}[X_i]=\mathbb{E}[Y_i]$ and alternatiave hypothesis $\mathbb{E}[X_i]\neq\mathbb{E}[Y_i]$. Now, we wish to estimate the null distribution using the permutation distribution. This is when I fail to understand: we then say under the null hypothesis, $X_i$ and $Y_i$ are identically distributed and then pool the r.v.s $X_i$ and $Y_i$ and find the permutation distribution.

Why do we know $X_i$ and $Y_i$ are identically distributed under the null? I mean, there mean is the same does not mean they share the same distribution at all. Help is appreciated. (Maybe I misunderstood what my textbooks are saying, if so, please point out my mistake.)


Why do we know $X_i$ and $Y_i$ are identically distributed under the null?

We don't know it -- it's an assumption -- but it's an easily justifiable one in many situations.

For example, imagine we draw subjects from a common pool and the new treatment is the old treatment with a small modification; and we believe that either the modification doesn't do anything (so the effect would be the same, that of the unmodified treatment) or it changes the mean (which is what we're interested in).

If the modification to the standard treatment doesn't do anything, the labels identifying which treatment a subject received are arbitrary, which is why you can permute them without it changing the distribution of the test statistic under the null hypothesis.

Now sometimes that assumption is less easy to justify. For example, you might think that under the null the means would be the same but the distribution would be expected to differ -- this is where the change in treatment does have an effect, but under the null it doesn't affect the mean.

Then the assumptions under which the permutation test operates won't hold (the observations won't be exchangeable under the null), and you you should consider whether you can find something more suitable to do (perhaps construct an interval for the mean treatment difference, for example).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.