Skip to main content

Questions tagged [false-discovery-rate]

An expected fraction of rejected null hypotheses that are falsely rejected, i.e. the fraction of significant findings that are actually not true. One method to control FDR in multiple testing is Benjamini-Hochberg procedure.

117 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9 votes
0 answers
582 views

How is it possible to control false discovery rate (FDR) without knowing the power and the prevalence of the nulls?

If we have p-value 0.05, to calculate probability of our discovery to be false positive, we need to use complex formula with prevalence (prior probability) and statistical power. If we have lots of ...
German Demidov's user avatar
7 votes
0 answers
144 views

A puzzling observation by Bradley Efron in his article in Science regarding Bayes’ Theorem in the 21st Century

Mr. Effron has published an interesting article in Science magazine with the enticing title "Bayes' Theorem in the 21st Century". The article is quite short and can be found here: http://web.ipac....
user8270077's user avatar
5 votes
0 answers
5k views

What is the difference between q-value and adjusted p-value (p.adjust)?

I know the p-value and I may know what FDR (false discovery rate) do and its goal. But I confuse between q-value (often known as FDR) and adjusted p-value (p.ajust ...
Aboubakar Ahamada's user avatar
5 votes
0 answers
788 views

Practical meaning of the PRDS property in Benjamini-Hochberg procedure under dependence

I am reading paper by Benjamini and Yekutieli (2001) on controlling FDR under dependence. My question is to figure out, in practical applications, whether the PRDS property is fulfilled in a given ...
January's user avatar
  • 7,567
5 votes
1 answer
1k views

How to estimate False Discovery Rate from p-value distribution?

I have learned many models and I calculated p-values for the cross-validation errors. I want to select significant models based on the false discovery rate (FDR). How can I estimate the FDR from p-...
user2806363's user avatar
  • 2,723
4 votes
0 answers
662 views

FDR and the the Benjamini-Hochberg Method

I am trying to understand the Benjamini-Hochberg Method for controlling the false discovery rate. Mathematically, if we are given with m hypothesis testing procedures, we sort the P-Values and reject ...
user205872's user avatar
4 votes
0 answers
439 views

The method of knock-offs by Barber & Candes for variable selection and FDR control

The knock-off method is a recent approach to variable selection and FDR control presented in two papers to be found here https://statweb.stanford.edu/~candes/papers/FDR_regression.pdf and here https://...
AlK's user avatar
  • 171
4 votes
0 answers
196 views

R : qvalue package - smoother or bootstrap method for pi0 estimation

I've a bunch of 500 p-values and I want to estimate the pi0 (proportion of true H0). When using the qvalue R package, two method can be used (smoother or bootstrap). Both giving very different results ...
Nicolas Rosewick's user avatar
4 votes
0 answers
584 views

Multiple testing correction alternatives

I have a data set with about 6,000 Fishers exact tests. If I want to do a multiple testing correction to control the FWER what are the advantages of one of these methods vs the other calculating a q-...
Brett Phinney's user avatar
3 votes
1 answer
457 views

running multiple t-tests across large dataset

I'm trying to work out if a t-test is the most appropriate in this situation: I have a data frame which looks like the one below but my data frame has aprox 37,000 rows. I'd like to run a t-test on ...
Chip 's user avatar
  • 173
3 votes
1 answer
123 views

Does a procedure controlling FDR at level $\alpha$ always reject at least as much as a procedure controlling FWE at $\alpha$?

I understand the false discovery rate (FDR) is weakly less than the familywise error rate (FWE), and FDR is thus a less stringent way to control for type 1 errors. However, will a procedure that ...
Golden_Ratio's user avatar
3 votes
0 answers
138 views

Correct Sample Space for Calculating P-Value

Suppose there are two variables, x and y, and we wish to calculate some test statistic (say the Pearson correlation), and estimate its significance. The pvalue is calculated by estimating the ...
Ami Tavory's user avatar
  • 4,590
3 votes
0 answers
897 views

p-value distribution and the applicability of different multiple testing correction methods

I came across this issue as I was analysing some genomic data sets looking for differential DNA methylation. I have generated p-values from the comparisons I am making, and am now looking into the ...
Reilstein's user avatar
  • 225
2 votes
0 answers
33 views

False rejection control techniques review

As far as I know, in multiple hypothesis testing scenarios there are methods to control for the False Discovery Rate (FDR) and the Family-Wise Error Rate (FWER) but I'm not aware of any method for ...
DaSim's user avatar
  • 460
2 votes
0 answers
455 views

Multiple comparisons version of Kolmogorov-Smirnov test

Suppose $\mathbf{y}_{i} $ and $ \mathbf{x}_{i}\,$ are length-$m$ vectors and $D_{i}$ is some arbitrary distribution $(i=1,...,N)$. I would like to conduct the following hypothesis test: $ H_{0}: \...
Mathew Carroll's user avatar
2 votes
0 answers
42 views

Correct to use FDR q-values for a small number of variables?

I would like to correct for multiple testing and I was considering using FDR q-values. In the original paper by Storey, they used this approach for genomewide data, where there are several thousands ...
Alex's user avatar
  • 181
2 votes
0 answers
84 views

Multiple Comparisons Correction for Multiple ANOVA models - Include only terms of interest?

I have a a biological dataset, in which I am fitting the following partial mixed ANOVA model (apologizes for the MATLAB syntax): ...
pdhami's user avatar
  • 335
2 votes
0 answers
318 views

Example of when FWER control should be preferred to FDR control

In the Benjamini & Hochberg paper (1995) which introduces the False Discovery Rate (FDR), the authors compare the FDR to the Family-Wise Error Rate (FWER): Often the control of the FWER is not ...
Simon Boehm's user avatar
2 votes
0 answers
62 views

Can FWER corrected p-values be called q-values?

This is a very naive question but I feel that many people in research do not have a very clear guidelines about what can or cannot be called a "q-value". I thought I had it right but I find so many ...
Maik's user avatar
  • 31
2 votes
0 answers
581 views

How permutation based FDR control methods take into account the correlation structure?

There are a number of permutations-based False Discovery Rate (FDR) control methods, such as SAM [1], dChip [2], Ge et al [3], Meinshausen [4], and Efron [5]. It's been said that in these methods, ...
Slouei's user avatar
  • 131
2 votes
0 answers
122 views

Is this a valid non-parametric approach for computing false discovery rate (FDR)?

Summarizing the method in the reference below as I understand it: For each of $N$ "gene sets" (where a gene set represents a single null hypothesis significance test), compute a test statistic $T_i$. ...
user3170906's user avatar
2 votes
0 answers
112 views

Estimating the False Reject Rate (FRR) of a classifier in production

I have trained a binary classifier which runs in production on remote distributed devices (which are out of my control). The model was trained on positive and negative samples, and I have chosen the ...
ShaharA's user avatar
  • 248
2 votes
0 answers
706 views

"FDR<0.05", or "FDR 0.05"?

I thought FDR is a threshold that we set, so we should speak about it without a < (or ) sign. For example, in a manuscript I ...
Vasily A's user avatar
  • 185
2 votes
1 answer
46 views

Multiple hypothesis test

I have some questions about multiple hypothesis test. Situation 1: We have 1000 p-values, all of them are less than 0.0 5. We may say there are 50 false positive (1000*0.05) in these p-values. ...
lyu su's user avatar
  • 29
2 votes
0 answers
875 views

multiple test correction procedure for survey data

I clustered ~ 20,000 students into three groups. Now, I also have answers from some of those students to ~ 100 survey questions. I would like to compare the differences between the clusters regarding ...
Vitomir Kovanovic's user avatar
2 votes
0 answers
683 views

Loss function for neural network to reduce false positives

I am trying to train a network to keep false positive as low as possible. I do not mind if I don't have extremely high accuracy, but I want the false positives to be within 5%. Are there any ...
stochastic_zeitgeist's user avatar
2 votes
0 answers
362 views

Storey q-values considerably lower than p-values (even for non-significant p)

I've been using the R package qvalue to estimate q-values based on p-values, and in most cases I have been observing results consistent with what one would expect ...
alan ocallaghan's user avatar
2 votes
1 answer
168 views

About controlling false positives

Doing multiple comparisons increases the number of false positive findings. In the Bonferroni method, the adjusted critical level is $\hat{\alpha} = \frac{\large\alpha}{N}$ where $N$ is the number of ...
user132576's user avatar
2 votes
0 answers
46 views

Multiple Comparisons with Different Tests

I have a couple of categorical variables, representing two characteristics, I will call them A and B, so that X1 is either A or not A (0/1) and X2 is either B or not B (0/1). In addition, I have many ...
user3275222's user avatar
2 votes
0 answers
82 views

$P$-value correction for ANOVA on repeated simulation

First of all, I would like to ask you if you know good literature that explains the theoretical underpinnings of multiple testing. Although I think I understand pretty well the basic principles, I ...
pierolefou's user avatar
2 votes
0 answers
157 views

fdr correction for omics data

I have a experiment design with 2 factors and I have 850 genes from this experiment. I want to look at individual genes before i go to multivariate dimension reduction . So I plan to fit a two way ...
statstar's user avatar
2 votes
0 answers
479 views

How is plot of p-values distribution related to calculation of the false discovery rate?

I'm running t-tests for all combinations of pairwise comparisons of average length of stay for patients among 30 hospitals. My first approach was to use a Bonferroni corrected significance level of ....
RobertF's user avatar
  • 6,124
2 votes
0 answers
66 views

Null distribution of the maximum and FDR control

Suppose we have a series of test statistics $T_{1},...,T_{m}$ each corresponding to a null hypothesis $H_{0}^{j}: \theta_{j} = 0$. Let $H_{0}: \theta_{1} = \ldots = \theta_{m} = 0$ denote the ...
gakn's user avatar
  • 86
2 votes
0 answers
90 views

Adjusted p-values across hypotheses with different power?

Simple question with no simple answer. Let's say I compute 100 p-values from 100 hypotheses that have similar power (i.e. similar sample size). Now let's say I produce 1000 p-values from a series of ...
user2815185's user avatar
2 votes
0 answers
302 views

Permutation test of meta-analysis : correlation coefficient

I am going to estimate False Discovery Rate using permutation tests. To my knowledge, several R packages are applicable for multiple testing. I have several independent datasets for meta-analysis. ...
chae's user avatar
  • 21
2 votes
0 answers
232 views

Confused by Benjamini-Hochberg-like false detection rate calculation in paper

I'm trying to figure out where the false detection rate correction in a genetics paper is coming from and failing. The paper in question is looking at a sequencing method where every sample is read ...
Jack Aidley's user avatar
1 vote
1 answer
33 views

How to apply FDR correction, and which method is most appropriate?

So I am a little confused with the application of q-values or more broadly, FDR correction (BH or BKY). I have run multiple linear mixed effect models to assess the impact of medication on several ...
John's user avatar
  • 11
1 vote
0 answers
36 views

Knockoff R package removing ALL variables based on LASSO, Why?

I am simulating a data set from a linear regression model and selecting the variables using LASSO (glmnet). The selection works relatively well with ...
Jack's user avatar
  • 11
1 vote
1 answer
101 views

Adjusting for multiple comparisons after Kruskal-Wallis

I am comparing biomarker levels (50 total) obtained from immunoassay in a cohort divided into 3 groups (control vs. inactive vs. active). I want to use the Kruskal-Wallis test with Dunn’s test to ...
NG0429's user avatar
  • 11
1 vote
1 answer
63 views

FDR correction - different number of p-values and models/hypotheses being tested

I have $3$ predictors: $1$ binary (prenatal alcohol exposure) $1$ categorical with $4$ categories (drinking patterns e.g., light stable, heavy reducer) $1$ continuous (total drinks consumed during ...
Emma's user avatar
  • 11
1 vote
0 answers
15 views

Should one correct for multiple comparison for secondary outcomes in RCTs or clinical trials?

This might a simple question answered elsewhere, but it comes up often in NHST discussions and I've heard conflicting statements. Imagine a metanalysis of 50 trials on 100,000 patients total to ...
DRG's user avatar
  • 323
1 vote
0 answers
98 views

Algorithm Validation and Alternative Approaches for FDR Control in Combined Datasets

We are currently working on a project involving multiple datasets reanalyzed with the same pipeline, where all data points (patient feature) have been filtered at 1% False Discovery Rate (FDR). As ...
ypriverol's user avatar
1 vote
0 answers
15 views

Calculating FDR in batches while taking into account the total number of tests

I am running a large number (potentially hundreds of millions - billions) null hypothesis significance tests (specifically a Poisson test, but the question is general). I would like to apply to them ...
user1614062's user avatar
1 vote
0 answers
14 views

FDR Correction needed on a pixel-wise comparison?

I have a question, which twirls my mind but I can't find a robust basis to answer it! I have a time-frequency data (let's say 5 columns x 5 rows) for each participant in two different groups. I have ...
KhonsKhandr's user avatar
1 vote
0 answers
28 views

How to Determine the Number of Independent Tests When Considering Multiple Testing

I have a question regarding how to determine the number of independent tests when performing multiple testing corrections. Suppose my research hypothesis is whether age is related to cognition decline ...
zjppdozen's user avatar
  • 347
1 vote
0 answers
88 views

Can the critical P-value returned by the Benjamini-Krieger-Yekutieli (BKY) procedure be greater than the false discovery rate?

If I use the Benjamini-Krieger-Yekutieli (BKY) procedure for an FDR correction, is it possible for the critical P-value returned to be greater than the desired false discovery rate? I just tried ...
Joel's user avatar
  • 61
1 vote
0 answers
24 views

Exploratory factor analysis (EFA) for transcriptomic data

I am doing an EFA for transcriptomic data (n=202, p=190). I did log-transformed the data because of skewness. My question is, do I have to do false discovery rate (FDR) analysis at all prior to my EFA....
Shehab's user avatar
  • 11
1 vote
0 answers
69 views

How do I use Benjamini-Hochberg procedure for post-test segmentation in A/B-tests correctly?

I am currently working on evaluating the results of an A/B-test. For this, I am also looking into some relevant post-test segments. Overall, I have no statistically significant difference between ...
Emily's user avatar
  • 11
1 vote
1 answer
303 views

P-value correction for multiple Mann-Whitney tests, some of them being dependent

I have performed multiple comparisons using Mann-Whitney U tests, and want to correct the p-values to know which results are worth reporting. The structure of the data is as follows : 2 experiments, ...
alpagarou's user avatar
1 vote
1 answer
169 views

Adjust p-value of the all output or just the variable of interest?

I am running linear mixed effect models using the lme4 package in R with the following output. ...
Jess H's user avatar
  • 11