34

I'm going to explain these in a few different ways because it helped me understand it. Let's take a specific example. You are doing a test for a disease on a group of people. Now let's define some terms. For each of the following, I am referring to an individual who has been tested: True positive (TP): Has the disease, identified as having the disease ...


30

You are correct in that sample size affects power (i.e. 1 - type II error), but not type I error. It's a common misunderstanding that a p-value as such (correctly interpreted) is less reliable or valid when the sample size is small - the very entertaining article by Friston 2012 has a funny take on that [1]. That being said, the issues with underpowered ...


24

Benjamini and Hochberg (1995) introduced the false discovery rate. Benjamini and Yekutieli (2001) proved that the estimator is valid under some forms of dependence. Dependence can arise as follows. Consider the continuous variable used in a t-test and another variable correlated with it; for example, testing if BMI differs in two groups and if waist ...


20

From your question and in particular your comments to other answers, it seems to me that you are mainly confused about the "big picture" here: namely, what does "positive dependency" refer to in this context at all -- as opposed to what is the technical meaning of the PRDS condition. So I will talk about the big picture. The big picture Imagine that you ...


20

This would obviously be an absolute nightmare to do in practice, but suppose it could be done: we appoint a Statistical Sultan and everyone running a hypothesis test reports their raw $p$-values to this despot. He performs some kind of global (literally) multiple comparisons correction and replies with the corrected versions. Would this usher in a golden ...


18

Great question! Let's step back and understand what Bonferroni did, and why it was necessary for Benjamini and Hochberg to develop an alternative. It has become necessary and compulsory in recent years to perform a procedure called multiple testing correction. This is due to the increasing numbers of tests being performed simultaneously with high ...


16

It so happens that by coincidence I read this same paper just a couple of weeks ago. Colquhoun mentions multiple comparisons (including Benjamini-Hochberg) in section 4 when posing the problem, but I find that he does not make the issue clear enough -- so I am not surprised to see your confusion. The important point to realize is that Colquhoun is talking ...


15

"Multiple comparisons" is the name attached to the general problem of making decisions based on the results of more than one test. The nature of the problem is made clear by the famous XKCD "Green jelly bean" cartoon in which investigators performed hypothesis tests of associations between consumption of jelly beans (of 20 different colors) and acne. One ...


13

@Dian breathe easy, it's pretty much not too difficult. So let's work from familiar territory to false discovery rate (FDR). First, I see that you have a bunch of outcomes, with a varying number of predictors. Someone who is more familiar with multivariate regression (i.e. multiple dependent variables, assuming possible correlations between errors of ...


13

Benjamini & Hochberg define false discovery rate in the same way that I do, as the fraction of positive tests that are false positives. So if you use their procedure for multiple comparisons you control FDR properly. It's worth noting, though, that there are quite a lot of variants on the B-H method. Benjamini's seminars at Berkeley are on Youtube, ...


12

Basically it is because controlling the FWER controls the probability of making a Type I error AT ALL and the FDR allows Type I Errors but controls how many of them you make in proporition to your true positives. The FDR has a higher power because it has a higher Type I error rate, which is a classical trade-off. This is just a short answer, but I think it ...


10

Indeed, @cardinal is quite right that the paper is as clear as it gets. So, for what it's worth, in case you do not have access to the paper, here's a slightly elaborated version of how Benjamini–Hochberg argue: The FDR $Q_e$ is the expected value of the proportion of false rejections $v$ to all rejections $r$. Now, $r$ is, obviously, the sum of false ...


10

I found this pre-print helpful in understanding the meaning. It should be said that I offer this answer not as an expert in the topic, but as an attempt at understanding to be vetted and validated by the community. Thanks to Amoeba for very helpful observations about the difference between PRD and PRDS, see comments Positive regression dependency (PRD) ...


10

Multiple comparisons corrections are intended to control the familywise error rate--or something like it--so they should be applied across a "family" of related hypothesis tests. In your first example, the overarching goal probably to determine whether Groups A and B differ. If you didn't control for multiple comparisons, you could trivially find an effect ...


9

First a to the point answer. Consider that $p_0$ is the (single test) $p$ value associated with value $z_0$ of the test statistic. The Benjamini-Hochberg FDR is computed in two steps ($N_0$ = # pvalues $\le$ $p_0$, $N$ = # pvalues): $\text{FDR }(p_0) = \frac{\quad p_0 \quad }{\frac{N_0}{N}}$ $\text{FDR }(p_i) = \min (\text{FDR}(p_i), \text{FDR}(p_{i+1}))$ ...


8

Your false discovery rate not only depends on the p-value threshold, but also on the truth. In fact, if your null hypothesis is in reality wrong it is impossible for you to make a false discovery. Maybe it's helpful to think of it like that: the p-value threshold is the probability of making false discoveries when there are no true discoveries to be make (...


7

Yes, this is possible, if the proportion of null hypotheses (which is estimated by the qvalue package based on your p-value distribution) is small and your test is powerful. Here's an example. Let's say you're testing 1000 hypotheses, and let's say 200 (20%) are actually null- this proportion is called $\pi_0$. Assume the qvalue package accurately estimates ...


7

Yes, these things are done quite often, at least in genetics. To address your specific points: This is quite commonly done, and I have personally reported results this way. Though, make sure to make it clear at what level of FDR you are reporting. Think of it akin to "marginal" significance; people may find it interesting, but they have to know what they're ...


7

I see people confusing this all the time, also in this forum. I think this is caused to a large extent because in practice Benjamini-Hochberg's procedure is used as a synonym of False Discovery Rate (and as a black-box for "adjusting" p-values as requested by reviewers for their papers). One has to clearly separate the FDR concept from Benjamini-Hochberg's ...


7

I think that you deliberately paint a pessimistic view of science produced by statistics. Indeed, in my opinion, statistics is not just a set of tools providing p values. There is also a state of rigour, care and alertness about some possible effects involved in the procedure of scientific induction... and while to my mind, everything you state is roughly ...


7

This is a good question, but you have several concepts confused. Firstly, to answer your broader question, yes splitting p-values and performing correction on them separately is an often-performed and well known approach when you have prior information about the system you're studying. To see more examples of this and a proof for independent tests, see Lei ...


6

You can convert from a q-value distribution to a p-value distribution rather simply (indeed, it's easier than the other way around!). The way to do this in R is (explanation is in the comments): convert.qval.pval = function(qvalues) { # you need to know the estimate of pi0 used to create the q-value # that's the maximum q-value (or very, very close ...


6

The FDRtool computes q-values by estimating the null fraction empirically from the data. p.adjust() with method "fdr" uses the Benjamini-Hochberg approach which effectively assumes that the null fraction is always 1.0. This implies that the p.adjust p-values are stochastically higher than q-values, and equal only if empirically estimated null fraction by ...


6

My PhD thesis was on precisely the topic of how to best test for significant differences in EEG and I faced the same questions. I found the optimal method is to use a mass-univariate test for each electrode and time/frequency point independently. This may be a t-test, or ANOVA (as in your case a repeated measures ANOVA), or even simply the mean differences, ...


6

In order to aggregate the results of multiple studies you should rather think of making your results accessible for meta analyses. A meta analysis considers the data of the study, or at least its estimates, models study effects and comes to a systematical conclusion by forming some kind of large virtual study out of many small single studies. The individual $...


6

If I [individual researcher] have a guess of what the size of the effect I'm studying should be [...], should I adjust my $\alpha$ level until the FDR = .05? Should I publish results at the $\alpha=.05$ level even if my studies are underpowered and leave consideration of the FDR to consumers of the literature? I would definitely not try to adjust the $\...


6

The plot shows the false coverage rate with a numerical example. $400,000$ hypotheses are being tested. $H_0:\theta=0$ for all tests, against a two sided hypothesis. The test statistic for each hypothesis is indeed $Y_i$. All null hypotheses are false, since under $\theta \sim exp(3)$ then $P(\theta=0)=0$. The slant lines are the FCR-adjusted intervals. ...


6

Depending on how you look at it, low power can increase false positive rates in given scenarios. Consider the following: a researcher tests a treatment. If the test comes back as insignificant, they abandon it and move onto the next treatment. If the test comes back significant, they publish it. Let's also consider that the researcher will tests some ...


6

The famous seminal Benjamini & Hochberg (1995) paper described the procedure for accepting/rejecting hypotheses based on adjusting the alpha levels. This procedure has a straightforward equivalent reformulation in terms of adjusted $p$-values, but it was not discussed in the original paper. According to Gordon Smyth, he introduced adjusted $p$-values in ...


6

If you are using R and want use the method of Benjamini and Hochberg (1995) to control the FDR, then you can use: FDR <- p.adjust(p, method="BH") where $p$ is the vector of p-values to compute BH adjusted p-values. To control the FDR at any specified level, say 0.05, you choose those cases with FDR < 0.05 as discoveries. Note that Benjamini and ...


Only top voted, non community-wiki answers of a minimum length are eligible