4
$\begingroup$

Recently, my colleague encountered a problem while working with a dataset, df_a, which contains gene data. Each row represents a gene (generally there are thousands, but for simplicity, let's assume there are 1,000), and columns represent different samples that can be divided into two groups, A and B (50 samples for each group).

We utilized the t.test to calculate the p-value for each row and performed a correction. Assuming we use p.adjust < 0.05 and we got 30 positive results.

This brings up a question: how can we ensure these 30 results are not generated by random events? (maybe the question is a question?)

We designed a process using permutation tests to address this question. The steps are:

(1) For df_a, we randomly shuffled its columns labels, recalculated the p-value and p.adjust for each row, and counted the number of rows with p.adjust < 0.05, denoted as Ki.

(2) Repeat step 1 for 1,000 times to get K1, K2, K3, ..., K1000.

(3) Count the number of Ki that is greater than 30, denoted as n. Calculate n/1000. If the value is less than 0.05, we consider that these 30 results are not generated by random events.

As a programmer, I realized that there seems to be a problem mathematically, but I can't provide a rigorous proof to my colleagues to correct it. I hope to get your help.

Update

What I want to update is, I think this permutation cannot provide meaningful additional information. Consider when you shuffle the column labels, for each row, it is equivalent to mixing groups A and B together, and then randomly drawing two groups A* and B*. Basic statistical principles tell us that there should be no difference between these two groups. Therefore, when you apply t.test, all p.values < 0.05 are false positives, and the significant number is about 5%. When you apply p.adjust to these data (for example, using FDR), and again control p.adjust < 0.05, the Ki here will be (always) very small (close to 0) that it is impossible to deny any results.

I'm not a mathematician, but it's easy to see this through program simulation. My point is when the number of permutations is finite (for example: 1000), the mathematical expectation of Ki is a constant related to row of data, and this value is very small.

Please be sure to note the differences between our steps and the classic permutation test, they are very similar, but there are differences.

I don't know if this holds true in mathematics and how to prove it.

Improve this question
$\endgroup$
15
  • 1
    $\begingroup$ I don't follow this, because you state you "calculate the p-value for each row," whence "randomly shuffling its row labels" should only permute the p-values without changing anything else. What am I getting wrong about your procedure? $\endgroup$
    – whuber ♦
    Commented Dec 10 at 14:58
  • $\begingroup$ @whuber, the column names (labels) of our data contain the grouping information. We always conduct t.test based on the groups in the column names. If we randomly shuffle the column names, it is equivalent to randomly selecting the values of group A and group B from each row. $\endgroup$
    – zhang
    Commented Dec 10 at 15:35
  • $\begingroup$ "If the value is less than 0.05, we consider that these 30 results are not generated by random events." This seems to make little sense. Let's say the value is indeed less than 0.05, but your permutations produced an average number of 20 rows with p.adjust<0.05. This means that you should well expect around 20 meaningless results. OK, 30 is more than that, but this will for sure not tell you that all 30 are not random. By the way, depending on what exact p.adjust you use, there may be theory that helps you specifying how many random significances you should expect. $\endgroup$
    – Christian Hennig
    Commented Dec 10 at 15:57
  • 1
    $\begingroup$ " how can we ensure these 30 results are not generated by random events?" Generally, you can't. The most conservative p-value adjustment is Bonferroni (very conservative with so many tests), and if a gene remains significant after Bonferroni (or somewhat better Bonferroni-Holm) correction, you can be as sure as it gets that there is something beyond randomness going on for that gene. For all 30, no chance. $\endgroup$
    – Christian Hennig
    Commented Dec 10 at 16:02
  • 1
    $\begingroup$ So--where you wrote "randomly shuffled its row labels" did you really mean column labels? $\endgroup$
    – whuber ♦
    Commented Dec 10 at 16:43

4 Answers 4

Reset to default
9
$\begingroup$

There is no statistical method that will allow you to "ensure" that significant results are not false positives. However, there are well-established scientific methods. Those methods involve re-testing the interesting hypotheses, or hypotheses derived from them, using new experiments and data. There is a very low probability that a false positive result in the first (preliminary) dataset will be followed by a falsely confirmatory false positive in the second.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yeah, I agree your are right, In fact, I believe that the above process will not help control false positives, or prove that the results are reliable, but I cannot prove this. $\endgroup$
    – zhang
    Commented 2 days ago
3
$\begingroup$

A use of permutations in controlling family wise error rate is described in

Romano, Joseph P., and Michael Wolf. "Exact and approximate stepdown methods for multiple hypothesis testing." Journal of the American Statistical Association 100.469 (2005): 94-108.

it may have advantages over a Holm method which may be too conservative because it is a strong control method. (it's control is not only valid when all null hypotheses are true, which is a weak control, but also when there is a subset for which the null hypothesis is false)

In these stepwise methods you reject/accept hypotheses according to the ordered p-values being below a certain level or the test statistics being in a certain critical region.

Your method is maybe a bit extreme, you only consider the number of cases of p-values below 0.05* and reject/accept the entire set based on that single point in the empirical distribution of p-values. It might be worth considering a more general method.

* btw you could compute this without permutations, as this is just binomial distributed

Improve this answer
$\endgroup$
8
  • $\begingroup$ Hi, I am confused with "you could compute this without permutations, as this is just binomial distributed", Can you further explain this? $\endgroup$
    – zhang
    Commented 2 days ago
  • $\begingroup$ @zhang I might have made that comment too quick. The thought behind it is that p-values are uniform distributed, and the probability for a p-value below 0.05 is 5% and the number of such p-values follows a binomial distribution. But that thought ignores 1) that the distribution might not need to be exactly uniform (if the t-test is not entirely correct), and also 2) because you are using corrected p-values, and it seems like you observe very small raw p-values, a permutation test might not be able to estimate the p-values accurately. $\endgroup$
    – Sextus Empiricus
    Commented 2 days ago
  • $\begingroup$ There's actually an interesting subtle effect that might support the idea of performing permutations instead of the computation with the binomial distribution that I suggested. It is not unthinkable that differences for multiple genes are correlated within a single sample. This makes that according to the calculation that I suggested the 30 p-values below the 5% can seem like a lot, but with a permutation test, this might turn out to be just a small amount. $\endgroup$
    – Sextus Empiricus
    Commented 2 days ago
  • $\begingroup$ Thanks for your reply, In fact, our data scale might be larger, such as over ten thousand genes, among which dozens are significant under the threshold of p.adjust < 0.05. Here I have made some simplifications to illustrate the problem, which may not be very appropriate. I apologize for any possible misguidance and confusion. $\endgroup$
    – zhang
    Commented 18 hours ago
  • $\begingroup$ My point is that regardless of how much the data varies, when you shuffle the column labels as described above, the p.values you get should be uniformly distributed. After adjusting with p.adjust, there are very few significant p.adjust (this can be seen through the program), to the extent that this permutation test always gives positive results. $\endgroup$
    – zhang
    Commented 18 hours ago
3
$\begingroup$

Your core question is: "This brings up a question: how can we ensure these 30 results are not generated by random events?"

I should point out that both the t.test and p.adjust already do this. The t.test gives you the probability of getting a result at least as extreme as the one you got if the truth is that your two groups are the same. If this p-value is large, then it implies that the difference you are seeing for this one gene is not unexpected under the assumption that there is really no difference (i.e. this difference is just from chance). To decide on how small your p-value should be, you set an arbitrary significance level, such as 0.05, before you do your experiment. Setting it up this way allows you to say that if there is indeed no real difference, then if you repeat the experiment 100 times, you will get p-values higher than 0.05 only 5% of the time, giving you "confidence" suppose you compute a p-value less than 0.05 to declare that you found something statistically significant.

Now, this becomes problematic when you do 1000 tests simultaneously with the hope of finding "significant" things among many possible targets (i.e. genes). If you conduct 1000 tests, each at a significance level of 0.05, then you can reject 50 hypotheses on average by "random chance" even if there really aren't any differences between groups in all of the genes. This is where p.adjust comes in. Depending on the method you selected for p.adjust, it will control for false positives.

Edit: (1) Does the permutation test process above prove that false positive control (p.adjust) is reliable?

No. The permutation test is just another test. You could have done permutation tests instead of t.tests, for example. It doesn't allow you to infer anything about the quality of p.adjust as a method.

(2) Does the permutation test process above help further control false positives?

Yes. Any sort of decision rule that increases the chance that some hypothesis can be flipped from "reject Ho" to "fail to reject Ho" will naturally help control false positives, but they will also increase the risk of false negatives! For example, you can arbitrarily say that for each gene that was identified as significant, I'm going to flip a coin and if it is heads, then I will keep retain the result but if it is tails, I will switch it to not significant. That reduces false positive rate too.

Improve this answer
New contributor
George Box is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
5
  • $\begingroup$ Yes, I actually agree with your point of view, but there are some disagreements within us. We all admit that what you said is correct (p.adjust can control false positives). But my colleagues hold two views: (1) The permutation test process above can prove that false positive control (p.adjust) is reliable, or (2) The permutation test process above can help further control false positives. I disagree with this view. I believe that the above process does not provide additional information for false positive control (no help), but I can't do this mathematically. $\endgroup$
    – zhang
    Commented 2 days ago
  • 1
    $\begingroup$ (1) is wrong, (2) is right, but not without consequence. I'll add this to my answer to not make the explanation $\endgroup$
    – George Box
    Commented 2 days ago
  • $\begingroup$ Thank you for your answer, I fundamentally agree with your response, but the second point seems to not align with our situation. Please note that our permutation test is different from the classic approach. What we observe is not the difference between real data and permuted data, but the difference the number of p.adjust < 0.05. $\endgroup$
    – zhang
    Commented yesterday
  • $\begingroup$ when we run this permutation test, the value of Ki is always very small (the average value of Ki is less than 1 and very close to 0, and the number of times Ki is greater than 0 in 1000 permutations is only 5%, and these values are usually 1), no matter what kind of data you use, so unless it's a very extreme case (for example, the number of real data p.adjust < 0.05 is only one), this permutation test will always give a positive result. But if we known it can only deny the situation where the number of real data p.adjust < 0.05 is only one, so why would you need to run it? $\endgroup$
    – zhang
    Commented yesterday
  • 1
    $\begingroup$ Your permutation test only checks how stable your results are. You don't know the actual behavior of your p-values under the alternative, you only assume that they ought to be uniformly distributed under the null. The test does not really give more than what p.adjust already does, it just makes your result arbitrarily stricter. $\endgroup$
    – George Box
    Commented yesterday
1
$\begingroup$

Few thoughts:

  1. This brings up a question: how can we ensure these 30 results are not generated by random events?

Unless you already know which hypothesis is true (the null or alternative), no statistical method can tell you which one is true with 100% confidence, rather, we get probabilities.

  1. What I want to update is, I think this permutation cannot provide meaningful additional information

A permutation test to compare 2 groups will be more robust than a t-test, because the t-test generally relies on asymptotic assumptions (large sample size) for valid results (unless you have perfectly normally distributed data, which is almost never the case). In contrast, the permutation test makes less assumptions.

So you could view the permutation test as a robust sensitivity analysis. Or you could just always go with the permutation test since it is better than the t-test.

  1. When calculating the permutation test pvalue, you need to add 1 to the numerator and add 1 to the denominator, in order to avoid getting pvalues of 0. You should never get a pvalue = 0, and if you did, that would bias your multiplicity adjustment. Source.

  2. When comparing a gene between 2 groups using a permutation test, you should permute the group labels.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yeah, but our step doesn't seem to be the classic permutation test steps. In the classic permutation test, we shuffle labels and compare the position of the actual differences and after shuffle , but we didn't do this. Simply put, in each test, we actually performed a t.test and what we ultimately compared was the number of p.adjust < 0.05. $\endgroup$
    – zhang
    Commented yesterday

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.