In permutation test, why do we take the proportion of sampled permutations with value equal or larger than the observed value?

Question

The tutorial I followed explains permutation testing in an intuitive way. However, it has confused me in one specific part. Why do we take as p-value the proportion/probability of permutation with test statistic equal or larger than the observed one?

In the tutorial, they explain the null and alternate hypotheses in terms of test statistics, i.e., in a mathematical way. And it makes sense.

It says, in short, we have a shampoo which claims to increase wool quality in sheep. So we have the following hypotheses.

Η_O = μ_treatment <= μ_control
Η_A = μ_treatment > μ_control

test_statistic = μ_treatment - μ_control

Here I'm guessing that the higher the μ the better the wool quality. So when they say that we need to check the probability of obtaining the observed test statistic, I understand that it is because we need to ensure the observed or greater value is not just random, that it is of significance. Here I understand that we need to take into account the greater values because the greater the test statistic, the more the claim of the shampoo company is verified.

I have understood this much. But then I see that all the permutation testing is done in this manner. Whether I understand the null hypothesis, everyone is using as the p-value the probability of obtaining the observed or greater value. Everyone is essentially applying this test in determining the effectiveness of "treatment", whether the treatment is vaccine, shampoo, diet, or really anything new. But I occasionally come across stuff that mention permutation test being a significance test. That means how likely is the observed value to occur by chance, right? Then why does it always include a treatment and a control group? Shouldn't this be applicable to any quantifiable value? For example, assume that I have a mathematical system for determining the attractiveness of people. You give it as input the age and income of people and it returns as output their attractiveness. Could I apply permutation test to see if the attractiveness values are significant? In other words, could I use permutation test to check if my mathematical method is correct?

Answer 1

Why do we take as p-value the proportion/probability of permutation with test statistic equal or larger than the observed one?

A p-value is defined to be the probability, according to the statistical model, of getting a test statistic value at least as large as that observed when the null hypothesis is true. The “getting a test statistic value” is often written as “obtaining data”. In the situation of your permutations test the sample mean is serving as your test statistic and the proportion of permutations in which the mean is as great or greater than the observed mean estimates the probability of interest, the p-value. (Assuming you’ve done a random permutations test that randomly samples the permutations. If you’ve done a complete permutations test that examines all of the possible permutations then the proportion is the probability, not just an estimate.) Why do we do that? Why is that the case? Because that’s what a p-value is, and that’s how we get it.

But I occasionally come across stuff that mention permutation test being a significance test. That means how likely is the observed value to occur by chance, right?

No! A permutations test is a significance test, yes, but the result is not the probability of getting anything by chance. It is the probability according to the statistical model of getting a result as extreme or more extreme when the null hypothesis is true. The two bits that I’ve emphasised there are absolutely critical and your “how likely ... by chance” phrasing ignores them entirely. (You are not the first or only one to make that critical error!)

Read this chapter for a helpful account of the role of statistical models and the distinction between statistical inferences which take place using and within a statistical model and the real-world inferences that people naturally want to make: https://link.springer.com/chapter/10.1007/164_2019_286

Then why does it always include a treatment and a control group? Shouldn't this be applicable to any quantifiable value?

You have to have something to compare with. You might be able to specify a particular value for the null hypothesis, but the null hypothesis in a significance test usually takes the form of a difference and we need to find both parts of the difference from in the observations or experiment. That is a type of control. There are many different types of control that might be needed for a good scientific experiment in addition to that basic control, but they might be necessary for things outside of the statistical model.

Could I apply permutation test to see if the attractiveness values are significant? In other words, could I use permutation test to check if my mathematical method is correct?

You can apply a permutations test to many different forms of data for many situations, no matter how poorly designed the study might be (see this: http://www.stat.columbia.edu/~gelman/research/unpublished/power4r.pdf). However, I don’t think that your use of “mathematical method” makes sense in that context.

Answer 2

Why do we take as p-value the proportion/probability of permutation with test statistic equal or larger than the observed one?

This is contextual, and not all permutation tests work this way. In any hypothesis test there is a test statistic that is used to establish an ordering over all possible data outcomes, describing whether outcomes are more or less conducive to the alternative hypothesis. In the specific test you are using, greater values of the test statistic (which you have stated incorrectly) are more conducive to the alternative hypothesis. Consequently, the p-value is the probability of getting a test statistic that is equal to or greater than the observed value.

In other contexts you might have a test statistic that has the opposite ordering --- i.e., lesser values being more conducive to the alternative hypothesis--- and in this case the p-value would be the probability of getting a test statistic that is equal to or lesser than the observed value.

When undertaking a permutation test, the sole purpose of the randomised permutations is to estimate the true p-value of the test by simulation. This does not change the underlying logic of the hypothesis test, and it still respects the evidentiary ordering induced by the test statistic.