Which Adjusted P value is the most accurate one (“fdr”,“none”,“bonferroni”)

I was wondering which method among "fdr","none","bonferroni" is the most accurate one to calculate the adjusted p-value for multiple comparison tests. Since when I define

summary(glht(fit1.2,linfct=mcp(Ethnic.Sex=cMat)),test=adjusted("none"))


I will have several significant p-values, but when I use Bonferroni and fdr the p-values are not significant. So any advice would be highly appreciated?

• Bonferroni is very conservative. FDR deals how many errors are allowed, a different criterion compared to maximizing the probability of no errors. – Michael R. Chernick Aug 15 '17 at 14:44
• As @Michael Chernick argues these procedures have different goals: Bonferonni controls the Familiy Wise Error Rate (FWER) and FDR controls False Discovery Rate (FDR). – user83346 Aug 15 '17 at 15:02
• It might help if you could include in your question the number of comparisons you are performing and provide some idea of the scale of your data set. – EdM Aug 15 '17 at 15:29

The idea behind these corrections is that the the chance of getting a false positive for several test say 20 at an $\alpha=0.05$ is 1 in 20. Out of 20 comparisons one has a strong possibility of being a false positive.

"none" says don't correct the test and use $\alpha = 0.05$.

The Bonferroni correction is the most conservative measure of the three you mentioned where the adjusted significance level is $\alpha/m$ where $m$ is the number of hypotheses.

FDR or False Discovery Rate correction in R is an alias for the 'BH' Benjamini & Hochberg correction. This method ranks the unadjusted p-values and the new pvalue is less than or equal to $\alpha * rank / m$ where $m$ is the number of hypotheses.

As a rule of thumb, in terms of a false positive I would order them as follows (lowest to highest):

bfr $<$ fdr $<$ none

You can do more research into each and learn about the family wise error rate vs the false discovery rate.

• thanks so much for your great advice. But does it really make sense to not adjust the p value (none) when we have several hypotheses to test? For example, I have two-way ANCOVA and I want to do the multiple comparison tests between levels. Do you think, I should use "none" and not adjust the p-value? – joe Aug 15 '17 at 15:14
• Also, why adjusted Bonferroni p-values are usually 1? – joe Aug 15 '17 at 15:22
• How many comparisons are being made? If there are many factors $m$ could be large and the significance level would be very conservative which would result in high p-values for small differences. I wouldn't go with none if the Bonferroni looks too conservative I'd switch to BH (fdr) correction method. – Matt L. Aug 15 '17 at 15:54
• @ Matt L., thanks so much, I have a variable with 2 levels and the second variables with 4 levels. So it would be around 28 comparisons. – joe Aug 15 '17 at 16:02
• I think you forgot some rather important conditionals on your sta – Björn Aug 15 '17 at 17:37