Intuitively, the numerator in the pvalue calculation for 2-sided permutation test involves counting the number of permuted values more extreme (either larger or smaller) than the true value.

If the empirical null distribution is centered at zero and symmetric, the numerator is found by simply counting the number of permuted statistics that have an equal or greater absolute value than the true value.

If the empirical null distribution is not centered at 0 but still symmetric, then intuitively, the counting described above can only occur after mean-centering the distribution. However, I'm not sure how to handle the situation where the null distribution is not symmetric. For example, if your statistic is the ratio between two variables, then the null might be centered at 1 and have a range between 0 and inf. In this specific case, you can make the distribution symmetric by taking log(ratio), but what should be done if the distribution is some completely screwy shape not associated with any normalization transform?

In other words, if the definition of "extreme" differs between both tails of the distribution, what is the right way to calculate the pvalue of the permutation test? It doesn't seem valid to simply pretend like both sides of the distribution use the same ruler, which happens when you just do pNumerator = sum[ abs(Permutations) >= abs(trueValue) ]

  • $\begingroup$ Besides specifying a test statistic you will need to clarify what you want to count as "at least as extreme". In may cases a good choice is obvious but in some cases there may not be a single one-size-fits-all answer-- it will come down to your needs and purposes. How do you want to measure deviation from the null? $\endgroup$
    – Glen_b
    Aug 6, 2018 at 9:27
  • $\begingroup$ No specific case in mind - it's a theoretical q. But, it seems to me that the numerator for the 2-tailed pvalue calculation I see everywhere might assume symmetry: +5 units above the mean (regardless of what the statistic is) is just as "extreme" as -5 below the mean. I'm sure I'm missing something, but I thought that perm tests shouldn't have such an assumption about the null distribution. Therefore, I thought there might be a natural way of dealing with it, if indeed my concern was real. $\endgroup$
    – Andy S
    Aug 6, 2018 at 10:09
  • $\begingroup$ There's nothing wrong with saying 5 above the mean is as extreme as 5 below even if the distribution isn't symmetric; if that's an appropriate measure of distance for your purpose why would it imply you think the distribution is symmetric? $\endgroup$
    – Glen_b
    Aug 6, 2018 at 11:37
  • $\begingroup$ I guess my thinking is; in the case of a ratio between two variables, the distribution is skewed so that one half of the distribution ranges from 1 to infinity, while the other ranges only from 1 to 0. So 1 - .01 is more extreme than 1 + .01. This particular case is fine because you can log-transform, but i wondered if a situation exists where 1. tails are not in the same range, 2. you want a 2-tailed test, and 3. no obvious transform exists to fix the ranges. But your comments suggest that if such a situation exists, I'm probably just not using an appropriate measure of distance, right? $\endgroup$
    – Andy S
    Aug 6, 2018 at 22:47
  • 1
    $\begingroup$ I think I understand what you're saying. Thanks. I think the bottom line is that the numerator calculation isn't so much assuming symmetry as it is assuming that both tails of the distribution use the same measuring stick/have the same range. If they don't, then /something/ needs to be done so that both tails use the same ruler for the calculation to be valid, but that thing depends on the specific distribution and there is no "more general formula" for the numerator that works every time regardless of whether the tails have different ranges or use different rulers. $\endgroup$
    – Andy S
    Aug 7, 2018 at 6:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.