How to permute p-values?

Question

Recently, my colleague encountered a problem while working with a dataset, df_a, which contains gene data. Each row represents a gene (generally there are thousands, but for simplicity, let's assume there are 1,000), and columns represent different samples that can be divided into two groups, A and B (50 samples for each group).

We utilized the t.test to calculate the p-value for each row and performed a correction. Assuming we use p.adjust < 0.05 and we got 30 positive results.

This brings up a question: how can we ensure these 30 results are not generated by random events? (maybe the question is a question?)

We designed a process using permutation tests to address this question. The steps are:

(1) For df_a, we randomly shuffled its columns labels, recalculated the p-value and p.adjust for each row, and counted the number of rows with p.adjust < 0.05, denoted as Ki.

(2) Repeat step 1 for 1,000 times to get K1, K2, K3, ..., K1000.

(3) Count the number of Ki that is greater than 30, denoted as n. Calculate n/1000. If the value is less than 0.05, we consider that these 30 results are not generated by random events.

As a programmer, I realized that there seems to be a problem mathematically, but I can't provide a rigorous proof to my colleagues to correct it. I hope to get your help.

Update

What I want to update is, I think this permutation cannot provide meaningful additional information. Consider when you shuffle the column labels, for each row, it is equivalent to mixing groups A and B together, and then randomly drawing two groups A* and B*. Basic statistical principles tell us that there should be no difference between these two groups. Therefore, when you apply t.test, all p.values < 0.05 are false positives, and the significant number is about 5%. When you apply p.adjust to these data (for example, using FDR), and again control p.adjust < 0.05, the Ki here will be (always) very small (close to 0) that it is impossible to deny any results.

I'm not a mathematician, but it's easy to see this through program simulation. My point is when the number of permutations is finite (for example: 1000), the mathematical expectation of Ki is a constant related to row of data, and this value is very small.

Please be sure to note the differences between our steps and the classic permutation test, they are very similar, but there are differences.

I don't know if this holds true in mathematics and how to prove it.

Answer 1

There is no statistical method that will allow you to "ensure" that significant results are not false positives. However, there are well-established scientific methods. Those methods involve re-testing the interesting hypotheses, or hypotheses derived from them, using new experiments and data. There is a very low probability that a false positive result in the first (preliminary) dataset will be followed by a falsely confirmatory false positive in the second.

Answer 2

A use of permutations in controlling family wise error rate is described in

Romano, Joseph P., and Michael Wolf. "Exact and approximate stepdown methods for multiple hypothesis testing." Journal of the American Statistical Association 100.469 (2005): 94-108.

it may have advantages over a Holm method which may be too conservative because it is a strong control method. (it's control is not only valid when all null hypotheses are true, which is a weak control, but also when there is a subset for which the null hypothesis is false)

In these stepwise methods you reject/accept hypotheses according to the ordered p-values being below a certain level or the test statistics being in a certain critical region.

Your method is maybe a bit extreme, you only consider the number of cases of p-values below 0.05* and reject/accept the entire set based on that single point in the empirical distribution of p-values. It might be worth considering a more general method.

---

* btw you could compute this without permutations, as this is just binomial distributed

Answer 3

Your core question is: "This brings up a question: how can we ensure these 30 results are not generated by random events?"

I should point out that both the t.test and p.adjust already do this. The t.test gives you the probability of getting a result at least as extreme as the one you got if the truth is that your two groups are the same. If this p-value is large, then it implies that the difference you are seeing for this one gene is not unexpected under the assumption that there is really no difference (i.e. this difference is just from chance). To decide on how small your p-value should be, you set an arbitrary significance level, such as 0.05, before you do your experiment. Setting it up this way allows you to say that if there is indeed no real difference, then if you repeat the experiment 100 times, you will get p-values higher than 0.05 only 5% of the time, giving you "confidence" suppose you compute a p-value less than 0.05 to declare that you found something statistically significant.

Now, this becomes problematic when you do 1000 tests simultaneously with the hope of finding "significant" things among many possible targets (i.e. genes). If you conduct 1000 tests, each at a significance level of 0.05, then you can reject 50 hypotheses on average by "random chance" even if there really aren't any differences between groups in all of the genes. This is where p.adjust comes in. Depending on the method you selected for p.adjust, it will control for false positives.

Edit: (1) Does the permutation test process above prove that false positive control (p.adjust) is reliable?

No. The permutation test is just another test. You could have done permutation tests instead of t.tests, for example. It doesn't allow you to infer anything about the quality of p.adjust as a method.

(2) Does the permutation test process above help further control false positives?

Yes. Any sort of decision rule that increases the chance that some hypothesis can be flipped from "reject Ho" to "fail to reject Ho" will naturally help control false positives, but they will also increase the risk of false negatives! For example, you can arbitrarily say that for each gene that was identified as significant, I'm going to flip a coin and if it is heads, then I will keep retain the result but if it is tails, I will switch it to not significant. That reduces false positive rate too.

Answer 4

Few thoughts:

1. This brings up a question: how can we ensure these 30 results are not generated by random events?

Unless you already know which hypothesis is true (the null or alternative), no statistical method can tell you which one is true with 100% confidence, rather, we get probabilities.

2. What I want to update is, I think this permutation cannot provide meaningful additional information

A permutation test to compare 2 groups will be more robust than a t-test, because the t-test generally relies on asymptotic assumptions (large sample size) for valid results (unless you have perfectly normally distributed data, which is almost never the case). In contrast, the permutation test makes less assumptions.

So you could view the permutation test as a robust sensitivity analysis. Or you could just always go with the permutation test since it is better than the t-test.

3. When calculating the permutation test pvalue, you need to add 1 to the numerator and add 1 to the denominator, in order to avoid getting pvalues of 0. You should never get a pvalue = 0, and if you did, that would bias your multiplicity adjustment. Source.

4. When comparing a gene between 2 groups using a permutation test, you should permute the group labels.