I'm comparing $m$ p-values, where $m$ is very large. Every p-value comes from a two-sample $t$-test of two groups of size $n_1$ and $n_2$, where the $n_i$ are very small ($2$ or $3$). This leads to a $(m\times(n_1+n_2))$ data matrix $X$.

Thus I would like to employ a permutation test to assess the significance of my p-values. I've got two questions regarding this problem:

  1. If I permute the sample labels for every test (e.g. with $n_1=n_2=2$ this gives 24 different label assignments) and calculate the corresponding $t$-statistic, I can evaluate if the original label assignment leads to a significant result by counting the number of permutations that lead to a larger $t$-statistic. Do I still need to correct for multiple testing?

  2. Since the $n_i$ are very small, does it make sense to permute the test labels instead? I.e., I draw $B$ times a set of $n_1+n_2$ values, where I draw one value from each column of $X$. This gives a null distribution based on these $B$ sets and their corresponding $t$-statistics. If I now compare the original $t$-statistics to this distribution, do I need to correct for multiple testing?

  • $\begingroup$ With $n_1 = n_2 = 2$, you only have six label assignments. Permutation testing won't help you much in this case. It seems to me that you are likely to be much better off with false discovery rate - type analyses of the $m$ p-values (en.wikipedia.org/wiki/False_discovery_rate). Do you have any particular reason to assume your data is close enough to Normally distributed for a t-test with a sample of size 2 or 3 to be (approximately) valid? $\endgroup$ – jbowman Nov 6 '18 at 14:58

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.