I have some results from a genome wide association study, which basically fits $n$ linear models for one trait $y$ and $n$ genotypes $x_1, x_2, \cdots, x_n$:

\begin{align*} y &\sim b_{10} + b_{11} x_1 \\ y &\sim b_{20} + b_{21} x_2 \\ &\vdots \\ y &\sim b_{n0} + b_{n1} x_n \\ \end{align*}

after which we get $n$ p values: $p_1, p_2, \cdots, p_n$. Normally $n$ is in the order of $10^6$, and the $\alpha$ level is usually set at $5 \times 10^{-8}$ in the literature, roughly following the Bonferroni correction.

Now based on these p values we have spotted an interesting region for which we have built a series of different models and conducted around 10000 more tests, the problem is how do we determine the significance cutoff point this time?

Someone has suggested to set it at $5 \times 10^{-8} / 10^4 = 5 \times 10^{-12}$, but apparently this is not quite reasonable, the 10000 tests are only a small number compared to the millions of test conducted previously.


1 Answer 1


Assuming a large sample, I suggest using cross-validation.

  1. Randomly split your data set in two (or more) sub-sets
  2. Build your model on one sub-set of the data using the Bonferroni correction you suggested.
  3. Use another sub-set of the data to test your explorative hypotheses built on the first set of the data. Now use a new Bnonferroni correction, e.g. by $10^4$.

Exploring and testing the hypotheses on the same set of data is generally considered bad practice and capitalizing on chance. Treating the data set 'as if' two (or more) random samples had been taken from the same population allows exploring and testing hypotheses in accordance to accepted standards. It is by the virtue of large data sets that this is possible.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.