Skip to main content
55 votes
Accepted

FPR (false positive rate) vs FDR (false discovery rate)

I'm going to explain these in a few different ways because it helped me understand it. Let's take a specific example. You are doing a test for a disease on a group of people. Now let's define some ...
mkt's user avatar
  • 18.8k
34 votes
Accepted

What's the formula for the Benjamini-Hochberg adjusted p-value?

The famous seminal Benjamini & Hochberg (1995) paper described the procedure for accepting/rejecting hypotheses based on adjusting the alpha levels. This procedure has a straightforward equivalent ...
amoeba's user avatar
  • 106k
27 votes

The meaning of "positive dependency" as a condition to use the usual method for FDR control

From your question and in particular your comments to other answers, it seems to me that you are mainly confused about the "big picture" here: namely, what does "positive dependency&...
amoeba's user avatar
  • 106k
26 votes
Accepted

Why aren't multiple hypothesis corrections applied to all experiments since the dawn of time?

This would obviously be an absolute nightmare to do in practice, but suppose it could be done: we appoint a Statistical Sultan and everyone running a hypothesis test reports their raw $p$-values to ...
Matt Krause's user avatar
  • 21.2k
19 votes
Accepted

Proof/derivation for false discovery rate in Benjamini-Hochberg procedure

Intro Let us start with some notation: We have $m$ simple hypotheses we test, with each null numbered $H_{0,i}$. The global null hypothesis can be written as an intersection of all the local nulls: $...
Spätzle's user avatar
  • 4,002
16 votes

What's the formula for the Benjamini-Hochberg adjusted p-value?

First a to the point answer. Consider that $p_0$ is the (single test) $p$ value associated with value $z_0$ of the test statistic. The Benjamini-Hochberg FDR is computed in two steps ($N_0$ = # ...
Aditya's user avatar
  • 261
12 votes

How many p-value observations do you think are required before doing FDR correction

There are many contextual variables that should be considered before deciding to use any type of adjustment for multiplicity of comparisons. The direct answer to your question is therefore that there ...
Michael Lew's user avatar
  • 15.8k
11 votes
Accepted

When do we "stop" using multiple correction techniques?

Multiple comparisons corrections are intended to control the familywise error rate--or something like it--so they should be applied across a "family" of related hypothesis tests. In your first ...
Matt Krause's user avatar
  • 21.2k
10 votes

The meaning of "positive dependency" as a condition to use the usual method for FDR control

I found this pre-print helpful in understanding the meaning. It should be said that I offer this answer not as an expert in the topic, but as an attempt at understanding to be vetted and validated by ...
Jacob Socolar's user avatar
9 votes

False discovery rate in binary logistic regression analysis?

You definitely should apply multiple testing. There are many options available to you. The choice depends on what kind of conclusions you want to make and what risk you want to take for the wrong ...
Michael R. Chernick's user avatar
8 votes

Proof/derivation for false discovery rate in Benjamini-Hochberg procedure

Caveat: this is just a set of notes for the proof in Spätzle's answer, and nothing more. Please, read his answer for the main notations. Additional notations: $N_0\subset\{1,\dots,m\}$ are the indexes ...
Zen's user avatar
  • 24.4k
7 votes

What are the practical differences between the Benjamini & Hochberg (1995) and the Benjamini & Yekutieli (2001) false discovery rate procedures?

p.adjust is not misciting for BY. The reference is to Theorem 1.3 (proof in Section 5 on p.1182) in the paper: Benjamini, Y., and Yekutieli, D. (2001). The control of the false discovery rate in ...
John Maindonald's user avatar
7 votes

Why aren't multiple hypothesis corrections applied to all experiments since the dawn of time?

I think that you deliberately paint a pessimistic view of science produced by statistics. Indeed, in my opinion, statistics is not just a set of tools providing p values. There is also a state of ...
beuhbbb's user avatar
  • 5,053
7 votes
Accepted

Over-represented values in FDR-adjusted p-values

There is nothing wrong with these q-values. Correcting p-values (corrected p-values are often referred to as q-values) is a concept that's newer than the Benjamini-Hochberg (BH) procedure, which in ...
einar's user avatar
  • 4,272
7 votes

How to calculate FDR and Power?

If you are using R and want use the method of Benjamini and Hochberg (1995) to control the FDR, then you can use: FDR <- p.adjust(p, method="BH") where $p$ is ...
Gordon Smyth's user avatar
  • 13.1k
7 votes
Accepted

Why bother with Benjamini-Hochberg correction?

This is a good question, but you have several concepts confused. Firstly, to answer your broader question, yes splitting p-values and performing correction on them separately is an often-performed ...
Chris C's user avatar
  • 2,630
6 votes
Accepted

On the generality of using empirical FDR even for conservative p-value distributions

Yes, that's a reasonable approach in principle. It's exactly what a permutation test does: estimate the distribution of any function under the null, then compare that to the value observed in the ...
eric_kernfeld's user avatar
6 votes
Accepted

fpr, fdr and fwe for feature selection

Those are among the methods for univariate feature selection. 1.13.2. Univariate feature selection Univariate feature selection works by selecting the best features based on univariate ...
Firebug's user avatar
  • 19.4k
6 votes

Rejection threshold of the Benjamini-Hochberg procedure

As you sense, there is no fixed p-value cutoff for the Benjamini-Hochberg control of false discovery rate. The cutoff depends on the specific distribution of p-values among the $m$ hypotheses that you ...
EdM's user avatar
  • 93.8k
6 votes

Proof/derivation for false discovery rate in Benjamini-Hochberg procedure

Geometric interpretation The values of the different p-values $p_1,p_2,\dots, p_n$ are distributed in a hypercube and rejection occurs when the point falls inside a region. The case of 2 variables For ...
Sextus Empiricus's user avatar
6 votes

How many coin flips are needed to reliably know a coin of weight w is unfair?

There are various ways you could examine this problem analytically, but a typical way is to frame the problem as a hypothesis test for a stipulated probability for the coin. Suppose we let $X_1,X_2,...
Ben's user avatar
  • 127k
5 votes

FDR correction when tests are correlated

You're looking for the Benjamini-Yekutieli procedure: Benjamini, Yoav; Yekutieli, Daniel. The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29 (2001), no. 4, ...
Zoë Clark's user avatar
5 votes
Accepted

Combining False Discovery Rates (FDR)?

You are not testing hypotheses, but fishing for interesting findings. There is nothing wrong with that. Do not do things that are often demanded by those who cannot tell the difference between a ...
Michael Lew's user avatar
  • 15.8k
5 votes
Accepted

Which confidence interval adjustment should I do when using FDR p valures adjustment?

FDR testing is implicitly step-down testing — testing certain things only if other tests pass. However, confidence intervals aren’t typically constructed conditionally, so there is no parallel. In ...
Russ Lenth's user avatar
  • 20.5k
5 votes
Accepted

About Kruskal-Wallis, Mann-Whitney U and multiple comparisons correction

I think your approach makes sense, but I would recommend the Dunn test as a post-hoc test. (No citation on the opinion.) This test is implemented in R, and in SPSS. I suspect it is found in other ...
Sal Mangiafico's user avatar
5 votes
Accepted

How to correct p-values of two multiple regression models

It is good that you are concerned with the type I error rate of running multiple models. However, keep in mind that any kind of correction will come at the cost of an increased type II error rate: By ...
Frans Rodenburg's user avatar
5 votes

How many coin flips are needed to reliably know a coin of weight w is unfair?

We can simplify the power calculation by approximating the sample distributions as normal distributions with $$\sigma \approx \sqrt{pq/n}$$ approximation $pq \approx 0.5^2$ such that $$\sigma\approx \...
Sextus Empiricus's user avatar
5 votes
Accepted

Q-value for FDR correction in R stats package

P-values never use set significance levels as part of their computation. In fact, the lack of reliance on any binary cutoff is part of the attraction of p-values. In the case of ...
Gordon Smyth's user avatar
  • 13.1k
5 votes
Accepted

False discovery rate correction when all P value are equal

The Benjamini-Hochberg FDR calculation performed by p.adjust seems perfectly appropriate in your situation. There is no theoretical problem with tied p-values in ...
Gordon Smyth's user avatar
  • 13.1k
5 votes
Accepted

Dealing with many positive-depenent insiginicant results in Benjamini-Hochberg-procedure

Before getting tied up in questions of how to deal with the power-robbing effects of 'corrections' for multiplicity, you should first think about whether your inferences will be improved by such ...
Michael Lew's user avatar
  • 15.8k

Only top scored, non community-wiki answers of a minimum length are eligible