# Should I adjust across samples or across tests for multiple hypothesis testing?

I have 100 samples and for each sample, I will perform 10 tests(here I only show 5 tests and 3 samples for simplicity). I am aware that I am running into multiple hypothesis testing issues and the p-value should be corrected, however, I am confused about which direction I should apply adjustment?

That is: should I adjust for column-wise P-value where each sample has 10 tests(10 p-values will be used to adjust for each sample)? or should I adjust for row-wise p-value where all the sample has been tested on the same test(100 p-values will be used as input for each test)? or maybe should I just go both directions (unlist the p values into 10*100=1000 individual test?)

Thanks for any help in advance!

• Adjustment of p-values for multiple testing is both controversial and dependent on context. Do not assume that you should, or must, adjust, but give more detail on the nature of the inferences and the role of the data in your scientific program. Mar 19, 2020 at 20:23
• @flash_Meow -- did you find my answer helpful? Mar 22, 2020 at 20:09

I believe I misunderstood your question the first time around.

As pointed out by others, there is usually not a recipe for how or when to perform multiple comparison testing. In order to have a better understanding for what might be appropriate in your case, we would need more information about your study design and what constitutes a "test."

Suppose you have a reference sample, Sample0, which has a list of numbers associated with it. In a series of tests, you compare SampleA, SampleB, and SampleC to Sample0.

Because each test is different, it has different statistical assumptions, statistical power, different robustsness to deviations from those assumptions, and perhaps even a different statistical test for calculating a p-value. E.g. one might be a t-test, another could be an F-test, another a Chi-Squared test, etc.

In such a case, the tests do not form a family, but the samples do. Since the samples are drawn from hypothetical parent populations that have something in common, there is always a random chance that non-significant differences will test as significant, or visa-versa. I would personally apply something like the BH correction across samples.

Meanwhile, suppose the samples were a completely heterogeneous collection, say individual colonies of different animal species, but the tests were repeated measurements of heart rate over the course of a year or something. The p-value compares the colony heart rate between males and females. In this case, column-wise multiple comparisons might make more sense. We want to control for false positives in the tests within each colony. But the colonies themselves have completely different statistical makeups.

I hope that helps.

If the goal is to conserve the family wise error, you should consider each cell a unique test. So, if it makes a p-value it's a test. Though with over 1,000 tests, you ought to consider controlling the false discovery rate. In either case, it's still focused on each unique row and column combination.

• So practically speaking, I should control for FDR for 1000 tests. In R, it would simply be p.adjsut(list_of_1000_pvalues,method="fdr")? Mar 19, 2020 at 21:51