# How many tests can one do in a medical publication?

Suppose we are in the following setting. There are 10 covariates and 1 response along with some other data derived from those covariates.

One can do regression of 10 covariates to 1 response to test individual coefficients significance.(This constitutes 10 tests though they might be dependent upon each other.) One can do testing on data derived from the covariates. Say 10 different tests have been done.

If I am using $$\alpha=5\%$$, am I going to run into the issue of inflating type I error here? I would say expected number of type I error is upper bounded by 2 out of 20 tests total due to correlation. However, for $$\alpha=1\%$$, I think problem would be alleviated a bit here.

How many test can one do in medical research article? Similarly, how many tests can be done on the same data set assuming a thorough exploratory data analysis has been done?

Indeed, any time you report a table of p-values for either separate test results or regression coefficients based on a Type I error of $$\alpha=0.05$$, you should take into account the multiple comparisons problem. One of the most conservative methods to account for this error is to use the Bonferroni correction, for which $$\alpha^*=0.05/\#\mathrm{tests}$$. Hence, p-values need to be less than $$\alpha^*$$ to be considered significant. However, Bonferroni is very conservative. Another approach to correct for family-wise error rate is to employ the Benjamini-Hochberg (1995) false discovery rate (FDR) to the p-values. Another approach involves calculation of Storey q-values to address the positive FDR (pFDR). Lastly, the Westfall-Young method is another approach which is informative.