Terminology: The terms permutation and randomisation tests often seem to be used interchangably, but in some cases they have distinct meanings - with the meanings not always consistent. In this post by randomisation test I mean a test where we randomly permute the observations in our sample(s) to recompute some test statistic, with the ultimate goal of comparing our observed statistic with the distribution of recomputed statistics and obtaining a simulated p-value. It will not be possible to compute every permutation so in practice we randomly shuffle our data a 'large' number of times.
Summary of question(s): I would like to know
What the "null hypothesis" of a randomisation (permutation) test should be, and how the results of the test should be interpreted? In particular focussing on an example where the two populations have the same mean, but different variances. Can we use this test to test for specific things like a difference in means, or can we only test for whether the distributions are different?
How does the choice of null hypothesis and statistic affect exchangeability, if at all?
I would appreciate a walkthrough of how a randomisation test could be applied to the example I give below.
Motivating Example: (See related:1 2) Two populations with the same mean but different variances. Group A, 30 people taken from $N(0,1)$. Group B 10 people from $N(0,4)$. We use a randomisation (permutation) test to obtain a simulated p-value for the difference in means. We observe that the simulated p-value is less than 0.05 about 33% of the time.
Python Code
import numpy as np
np.random.seed(0)
# Let's shuffle the labels of the groups and calculate the new difference in means.
# We will discuss later whether this is valid and how to interpret this
# We will count how many times we get a "p-value" less than 0.05
small_pval = 0
# We will simulate taking samples from our two populations many times, and running a randomisation test on them.
N=1000
for i in range(N):
# Take samples from our two distributions
A = np.random.normal(loc=0,scale=1,size=30)
B = np.random.normal(loc=0,scale=6,size=10)
observed_diff = np.mean(A)-np.mean(B)
# Combine into one list
combined = np.concatenate((A,B),0)
differences_in_means = np.array([])
for i in range(1000):
np.random.shuffle(combined)
diff = np.mean(combined[:20]) - np.mean(combined[20:])
differences_in_means = np.append(differences_in_means, diff)
# How often do we see a difference as extreme as the one we observed initially
# Take absolute value for 2 tail but I think this is not very significant - :D
simulated_pvalue = np.mean(np.abs(differences_in_means) >= np.abs(observed_diff))
if simulated_pvalue < 0.05:
small_pval+=1
print("We found simulated p-values less than 0.05 ", 100*small_pval/N, "% of the time")
Running the code above, we would find simulated p-values <0.05, about 33% of the time. Now, the statistic we have 'tested' is difference in means, however we know that both distributions have the same mean. The first thing I am trying to grasp, is what does the simulated p-value actually mean, in particular what is the associated null hypothesis for these p-values?
I could say something like: My null hypothesis is that the mean of population A = mean population B, I permute the labels many times, and find that the difference in means I have observed is very unusual (simulated p-value is smaller <0.05) and so I reject my null hypothesis and conclude (wrongly) that the means of the two populations are actually different. As the simulation above shows, if I repeated this, I would make this mistake about 33%, much more than the 0.05% we might expect.
I think that part of the problem here is exchangeability and the set up of the "null hypothesis." I think that here due to the difference in standard deviation, I cannot say that the labelling of group A and B is completely arbitrary and exchange them however I wish. However I think the null hypothesis above is not right for the randomisation test, but I have commonly seen people use the test this way - essentially as a direct replacement for the Student or Welch t-test. What should the null hypothesis be here to use a randomisation test? These quotes from Howell 3 seem relevant:
Our null hypothesis has nothing to do with parameters, but is phrased rather vaguely, as, for example, the hypothesis that the treatment has no effect on the how participants perform. That is why I earlier put "null hypothesis" in quotation marks. This is an important distinction. The alternative hypothesis is simply that different treatments have an effect. But, note that we haven't specified whether the difference will reveal itself in terms of means, or variances, or some other statistic. We leave that up to the statistic we calculate in running the test. That might be phrased a bit more precisely by saying that, under the null hypothesis, the score that is associated with a participant is independent of the treatment that person received
I need to say something about exchangeability. It applies to the null hypothesis sampling distribution--in other words, data are exchangeable under the null. Phil Good, for example, is a big fan of this term. He would argue that if the scores in one group have a higher variance than the other, then the data are not exchangeable and the test is not valid. BUT, if the hypothesis being tested is that treatments have no effect on scores, then under the null hypothesis why would one set of scores have a higher variance other than by chance? The problem is that we have to select the statistic to test with care. We normally test means, or their equivalent, but we also need to consider variances, for example, because that is another way in which the treatment groups could differ. If we are focussing on means, then we have to assume exchangeability including variance. But we need to be specific. So much for that little hobby horse of mine.
Howell also shares the following extract from Edgington (1986).
Just as the reference set (read as "sampling distribution" for now) of data permutations is independent of the test statistics, so is the null hypothesis. A difference between means may be used as a test statistic, but the null hypothesis does not refer to a difference between means. The null hypothesis, no matter what test statistic is used, is that there is no differential effect of the treatments for any of the subjects. ... Thus the alternative hypothesis is that the measurement of at least one subject would have been different under one of the other treatment conditions. Inferences about means must be based on nonstatistical considerations; the randomization test does not justify them. (p. 531)
Based on this, I wonder, if instead my "null hypothesis" was that the two populations are from the same distribution and then I have a small simulated p-value I could conclude that because the difference in means we have observed is so unusual that the distributions of the two populations must be different. This also seems strange to me, we have made the correct conclusion that the populations are from different distributions where the statistic we are using is difference in means, even though the means are actually the same. I note also if this was the null hypothesis, then in this case we have very low power here (33%), and wonder if that is precisely because the statistic we are using is the difference in means.
If the 'correct' null hypothesis for a (two sample) randomisation test is "The two populations are the same" and the alternative is then "the two populations are not the same" would this mean:
- I have more freedom with respect to exchangeability, for example we now have exchangeability in the example above.
- The power (and type 1 error rate) of the permutation test may be quite sensitive to whatever statistic I decide to use (say difference in means or Welch's t).
- If we find a significant result using some statistic, say difference in means, all we can conclude from this is that the two populations are different, but we actually cannot conclude anything more (like the means are different).
In the post here, an example very similar to the above is considered. It is mentioned that the type 1 error rate is inflated, and this can be remedied by using the Welch t statistic instead of the difference in means. The problem that is mentioned is lack of exchangeability.
- Why does the Welch t-statistic have a lower type 1 error rate than just looking at the difference in means?
- If the null hypothesis is "The means of these two distributions are the same," then does using Welch t-statistic mean we have exchangeability? If so why? The answer of BruceET here says
A permutation test with the Welch t statistic as metric treats samples with unequal variances as exchangeable (even if data may not be normal). 3) Am I right in thinking if we want to specifically conclude something about the means, then we need to set up the null hypothesis carefully and choose our statistic carefully (to ensure exchangeability), but if we just want to conclude the samples are from different populations then we have more freedom?
A few references to help highlight my confusion
The answer to this similar question:
Strictly speaking, the null hypothesis is that the distributions are the same, not just that they have the same means. (If they had same means but difference variances, the test would have the wrong Type I error rate.)
whereas the answer to this question phrases the null hypothesis of the permutation test in terms of means.
In this copy of ThinkStats the permutation test is introduced, it seems to me that text is saying we can choose whatever statistic we are interested in, and see if there is a significant difference in it, and then make a conclusion about that statistic. However we have seen from the example above that is not right, we found that we had an 'extreme' difference in means even though the means of the two groups are the same.
One way to model the null hypothesis is by permutation; that is, we can take values for first babies and others and shuffle them, treating the two groups as one big group....
Choosing the best test statistic depends on what question you are trying to address. For example, if the relevant question is whether pregnancy lengths are different for first babies, then it makes sense to test the absolute difference in means, as we did in the previous section. If we had some reason to think that first babies are likely to be late, then we would not take the absolute value of the difference; instead we would use this test statistic.
This question, which seems relevant but I do not think addresses my exact questions, it does not mention exchangeability nor how the test should be “set up” with a null and alternate hypothesis.
This question is closest. The asker posts a comment on the top answer which is one of the things I am getting at, but their comment is unanswered.
I read the paper. Do I understand correctly that permutation test applied to a location test doesn't test the null of location well, but rather tests whether the distributions of the two groups are the same in the sense of their locations?
I am a novice statistician. As you can no doubt tell, I am having trouble even articulating my questions. If I have misrepresented anything in any of the links or quotes provided that is not my intention, it is just due to my own confusion, I would gladly accept any edits/comments. I have also tried to link to other relevant questions, I do not think my question is a duplicate, if I have overlooked something relevant I would be glad to have it brought to my attention.
