The tutorial I followed explains permutation testing in an intuitive way. However, it has confused me in one specific part. Why do we take as p-value the proportion/probability of permutation with test statistic equal or larger than the observed one?
In the tutorial, they explain the null and alternate hypotheses in terms of test statistics, i.e., in a mathematical way. And it makes sense.
It says, in short, we have a shampoo which claims to increase wool quality in sheep. So we have the following hypotheses.
Η_O = μ_treatment <= μ_control Η_A = μ_treatment > μ_control test_statistic = μ_treatment - μ_control
Here I'm guessing that the higher the μ the better the wool quality. So when they say that we need to check the probability of obtaining the observed test statistic, I understand that it is because we need to ensure the observed or greater value is not just random, that it is of significance. Here I understand that we need to take into account the greater values because the greater the test statistic, the more the claim of the shampoo company is verified.
I have understood this much. But then I see that all the permutation testing is done in this manner. Whether I understand the null hypothesis, everyone is using as the p-value the probability of obtaining the observed or greater value. Everyone is essentially applying this test in determining the effectiveness of "treatment", whether the treatment is vaccine, shampoo, diet, or really anything new. But I occasionally come across stuff that mention permutation test being a significance test. That means how likely is the observed value to occur by chance, right? Then why does it always include a treatment and a control group? Shouldn't this be applicable to any quantifiable value? For example, assume that I have a mathematical system for determining the attractiveness of people. You give it as input the age and income of people and it returns as output their attractiveness. Could I apply permutation test to see if the attractiveness values are significant? In other words, could I use permutation test to check if my mathematical method is correct?