# Questions tagged [false-discovery-rate]

An expected fraction of rejected null hypotheses that are falsely rejected, i.e. the fraction of significant findings that are actually not true. One method to control FDR in multiple testing is Benjamini-Hochberg procedure.

274 questions
Filter by
Sorted by
Tagged with
1 vote
12 views

### Should one correct for multiple comparison for secondary outcomes in RCTs or clinical trials?

This might a simple question answered elsewhere, but it comes up often in NHST discussions and I've heard conflicting statements. Imagine a metanalysis of 50 trials on 100,000 patients total to ...
1 vote
89 views

### Algorithm Validation and Alternative Approaches for FDR Control in Combined Datasets

We are currently working on a project involving multiple datasets reanalyzed with the same pipeline, where all data points (patient feature) have been filtered at 1% False Discovery Rate (FDR). As ...
1 vote
7 views

### Calculating FDR in batches while taking into account the total number of tests

I am running a large number (potentially hundreds of millions - billions) null hypothesis significance tests (specifically a Poisson test, but the question is general). I would like to apply to them ...
1 vote
69 views

### Estimate the false discovery proportion for a fixed number of discoveries

Suppose we are testing a large number ($p$) hypothesis tests. Most commonly, a correction is applied to control the false discovery rate (FDR) at a low value, say $\alpha$. But now imagine a want to ...
1 vote
10 views

### FDR Correction needed on a pixel-wise comparison?

I have a question, which twirls my mind but I can't find a robust basis to answer it! I have a time-frequency data (let's say 5 columns x 5 rows) for each participant in two different groups. I have ...
1 vote
20 views

### How to Determine the Number of Independent Tests When Considering Multiple Testing

I have a question regarding how to determine the number of independent tests when performing multiple testing corrections. Suppose my research hypothesis is whether age is related to cognition decline ...
1 vote
42 views

### Can the critical P-value returned by the Benjamini-Krieger-Yekutieli (BKY) procedure be greater than the false discovery rate?

If I use the Benjamini-Krieger-Yekutieli (BKY) procedure for an FDR correction, is it possible for the critical P-value returned to be greater than the desired false discovery rate? I just tried ...
24 views

### How to calculate the false discovery proportion from the p-value histogram?

I read the textbook https://web.stanford.edu/class/bios221/book/06-chap.html about the false discovery proportion and the p-value histogram. ...
197 views

### Pairwise comparison; Tukey HSD or FDR

I have a general question about running multiple pairwise comparisons. I did some genetic work in grad school, where we (I believe) utilized a statistical test (Student's T-test?) followed by a FDR-...
36 views

### False positive correction categorical data

I am trying to understand how to correct for chance categorisation as either impaired/not impaired across a range of measures. Briefly, I am looking to summarise prevalence statistics across ~40 ...
1 vote
21 views

### Exploratory factor analysis (EFA) for transcriptomic data

I am doing an EFA for transcriptomic data (n=202, p=190). I did log-transformed the data because of skewness. My question is, do I have to do false discovery rate (FDR) analysis at all prior to my EFA....
55 views

### False Discovery Rate FDR for multiple regression models on same response

I'm trying to work out how to apply FDR (any of the methods) to my analysis. I have 12 different models all regressing on the same response, with the same dataset. When I am adjusting the 'p-values' ...
81 views

### How does p-value affect FDR

Is it always true that a lower p-value threshold will lead to a lower FDR? To answer this question I would say that it is not always true, for example in the case when the distribution is not normal ...
1 vote
53 views

### How do I use Benjamini-Hochberg procedure for post-test segmentation in A/B-tests correctly?

I am currently working on evaluating the results of an A/B-test. For this, I am also looking into some relevant post-test segments. Overall, I have no statistically significant difference between ...
26 views

### Methods to control for false omission rate?

What methods are available to control for false omission rate in multiple comparisons, similar to the way Benjamini-Hochberg controls for false discovery rate? The goal is to filter out obvious ...
36 views

### Does it make sense to use a FDR below 0.01 when using a multiple testing correction technique?

If we have an FDR of 1% then this would mean that among all features called significant, 1% of these are actually not. So If I have 700 features that are significant, then 7 would actually not be. ...
46 views

### Harmonic mean of false positive and false discovery rates (analogous to F1)

F1 is the harmonic mean of recall (aka sensitivity, or true positive rate, TPR) and precision (aka positive predictive value, PPV). $\text{TPR} = \text{Pr(predicted:Pos | Pos)} =$ TP/P (wikipedia ...
1 vote
664 views

### False discovery rate correction when all P value are equal

If we are have calculated multiple P values from independent tests, we can correct for the False Discovery Rate by using: ...
334 views

### running multiple t-tests across large dataset

I'm trying to work out if a t-test is the most appropriate in this situation: I have a data frame which looks like the one below but my data frame has aprox 37,000 rows. I'd like to run a t-test on ...
1 vote
165 views

### P-value correction for multiple Mann-Whitney tests, some of them being dependent

I have performed multiple comparisons using Mann-Whitney U tests, and want to correct the p-values to know which results are worth reporting. The structure of the data is as follows : 2 experiments, ...
39 views

### Test for gaussian mixture fit when component assignment is known?

I have a process 𝑃 generating random variables X_1, ... X_n. From each of these I've sampled a set of samples ...
30 views

### False Discovery Rate pvalue correction method [duplicate]

I am trying different methods to correct a p-value from simulated data,namely the following "holm", "hochberg", "hommel", "bonferroni","BH", "BY&...
1 vote
133 views

### Adjust p-value of the all output or just the variable of interest?

I am running linear mixed effect models using the lme4 package in R with the following output. ...
53 views

### calculating expected value of false discovery rate - Benjamini-hochberg

I recently got a question that cannot seem to solve, as a beginner in the statistical field. assume we have 20 tests. 5 with the p-value of 1/330 5 with the p-value of 1/150 10 with the p-value of 1/...
1 vote
863 views

### Do the p-values produced by lmerTest::lmer need to be adjusted for multiple comparisons?

I am running a fairly simple linear mixed model with multiple observations per patient, comparing a lab value to a reference group (healthy control). Here's some fake data: ...
37 views

### How to adjusting p-values from results of trend analysis on water quality data

I have results from Mann Kendall test for trends at 100 sample locations for 5 different analytes at each location (i.e., 500 p-values). If I had the trend results from 500 sample locations for 1 ...
2k views

### Q-value for FDR correction in R stats package

I used FDR correction (i.e., Benjamin-Hochberg tests) from the R 'Stats' package, but I'm confused of the q value used in the package. The notations in the 'p.adjust' function does not include the ...
1 vote
40 views

### When running BH Procedure and there are different types of statistical tests, are they all included?

Let's say I'm running 100 statistical tests. Some of them are t tests, some are ANOVAs (and for each ANOVA subsequent post-hoc t tests should a significant result be obtained), some of them ...
1 vote
241 views

### FDR correction stronger than Bonferroni?

Is it possible to get a situation where, in a multiple-testing scenario, for some individual tests, the Benjamini&Yukatieli (2001)-correction ends up being more stringent than Bonferroni? I.e. ...
421 views

### Should I perform multiple testing correction (FDR) for all the P-values regardless if they derive from Ttest or Wilcoxon?

There are 100 participants and I want to test if 50 metabolites change before and after intervention (antiobesity drug). First, I check the normality for each variable, and then I perform a T-test or ...
33 views

### False rejection control techniques review

As far as I know, in multiple hypothesis testing scenarios there are methods to control for the False Discovery Rate (FDR) and the Family-Wise Error Rate (FWER) but I'm not aware of any method for ...
184 views

### Question about power of FDR and FWE

I understand that false discovery rate (FDR) is $\leq$ familywise error rate (FWE). I have read this means that controlling FDR is hence "more powerful" than controlling FWE. What does ...
86 views

### Does a procedure controlling FDR at level $\alpha$ always reject at least as much as a procedure controlling FWE at $\alpha$?

I understand the false discovery rate (FDR) is weakly less than the familywise error rate (FWE), and FDR is thus a less stringent way to control for type 1 errors. However, will a procedure that ...
1k views

### How many coin flips are needed to reliably know a coin of weight w is unfair?

I want to find out how many flips I need to flip a coin to reliably know that it is an unfair coin. The issue is that as the coin becomes closer to 50/50, the more false-negatives you will have if you ...
114 views

### Is the FDR linear-step up procedure Benjamini and Hochberg valid for empircal p-values from non-parametric tests (e.g., bootstrap)?

I have empirical p-values from bootstrapped hypothesis tests with 10,000 resamples each. Because there are a finite number of resamples, some of the p-values are "0", i.e., < 0.0001 with ...
74 views

### Analysis sense check: preregister planned contrasts which we wish to conduct even if associated ANOVA is nonsignificant?

I could use a sense check on my analysis plan for an experimental study which has not yet been run; I will preregister it once I am confident that it is reasonable. There are seven between-...
1 vote
47 views

### Is there a way to adjust FDR threshold using correlation across tests (aka, number of estimated independent tests, like in Bonferroni correction)

For Bonferroni correction, correlations can be used to inform correction of multiple testing using the spectral decomposition of matrices. For example, in GWAS, the threshold is established to be 5e-8 ...
381 views

### How do I correct for multiple testing when I have multiple models?

I have 12 concentrations that I would like to compare between 2 groups, some subjects have 2 measures. I know the concentration is also depending on age and sex so I added those to the linear model ...