Questions tagged [false-discovery-rate]
An expected fraction of rejected null hypotheses that are falsely rejected, i.e. the fraction of significant findings that are actually not true. One method to control FDR in multiple testing is Benjamini-Hochberg procedure.
274
questions
1
vote
0
answers
12
views
Should one correct for multiple comparison for secondary outcomes in RCTs or clinical trials?
This might a simple question answered elsewhere, but it comes up often in NHST discussions and I've heard conflicting statements. Imagine a metanalysis of 50 trials on 100,000 patients total to ...
- 291
1
vote
0
answers
89
views
Algorithm Validation and Alternative Approaches for FDR Control in Combined Datasets
We are currently working on a project involving multiple datasets reanalyzed with the same pipeline, where all data points (patient feature) have been filtered at 1% False Discovery Rate (FDR). As ...
- 61
1
vote
0
answers
7
views
Calculating FDR in batches while taking into account the total number of tests
I am running a large number (potentially hundreds of millions - billions) null hypothesis significance tests (specifically a Poisson test, but the question is general). I would like to apply to them ...
- 121
1
vote
1
answer
69
views
Estimate the false discovery proportion for a fixed number of discoveries
Suppose we are testing a large number ($p$) hypothesis tests. Most commonly, a correction is applied to control the false discovery rate (FDR) at a low value, say $\alpha$. But now imagine a want to ...
- 1,617
1
vote
0
answers
10
views
FDR Correction needed on a pixel-wise comparison?
I have a question, which twirls my mind but I can't find a robust basis to answer it!
I have a time-frequency data (let's say 5 columns x 5 rows) for each participant in two different groups. I have ...
- 11
1
vote
0
answers
20
views
How to Determine the Number of Independent Tests When Considering Multiple Testing
I have a question regarding how to determine the number of independent tests when performing multiple testing corrections.
Suppose my research hypothesis is whether age is related to cognition decline ...
- 45
1
vote
0
answers
42
views
Can the critical P-value returned by the Benjamini-Krieger-Yekutieli (BKY) procedure be greater than the false discovery rate?
If I use the Benjamini-Krieger-Yekutieli (BKY) procedure for an FDR correction, is it possible for the critical P-value returned to be greater than the desired false discovery rate? I just tried ...
- 61
0
votes
0
answers
24
views
How to calculate the false discovery proportion from the p-value histogram?
I read the textbook https://web.stanford.edu/class/bios221/book/06-chap.html about the
false discovery proportion and the p-value histogram.
...
- 63
4
votes
1
answer
197
views
Pairwise comparison; Tukey HSD or FDR
I have a general question about running multiple pairwise comparisons. I did some genetic work in grad school, where we (I believe) utilized a statistical test (Student's T-test?) followed by a FDR-...
- 41
0
votes
0
answers
36
views
False positive correction categorical data
I am trying to understand how to correct for chance categorisation as either impaired/not impaired across a range of measures.
Briefly, I am looking to summarise prevalence statistics across ~40 ...
- 23
1
vote
0
answers
21
views
Exploratory factor analysis (EFA) for transcriptomic data
I am doing an EFA for transcriptomic data (n=202, p=190). I did log-transformed the data because of skewness.
My question is, do I have to do false discovery rate (FDR) analysis at all prior to my EFA....
- 11
0
votes
0
answers
55
views
False Discovery Rate FDR for multiple regression models on same response
I'm trying to work out how to apply FDR (any of the methods) to my analysis. I have 12 different models all regressing on the same response, with the same dataset.
When I am adjusting the 'p-values' ...
- 1
0
votes
0
answers
81
views
How does p-value affect FDR
Is it always true that a lower p-value threshold will lead to a lower FDR?
To answer this question I would say that it is not always true, for example in the case when the distribution is not normal ...
- 101
1
vote
0
answers
53
views
How do I use Benjamini-Hochberg procedure for post-test segmentation in A/B-tests correctly?
I am currently working on evaluating the results of an A/B-test. For this, I am also looking into some relevant post-test segments.
Overall, I have no statistically significant difference between ...
- 11
0
votes
0
answers
26
views
Methods to control for false omission rate?
What methods are available to control for false omission rate in multiple comparisons, similar to the way Benjamini-Hochberg controls for false discovery rate?
The goal is to filter out obvious ...
- 253
0
votes
0
answers
36
views
Does it make sense to use a FDR below 0.01 when using a multiple testing correction technique?
If we have an FDR of 1% then this would mean that among all features called significant, 1% of these are actually not. So If I have 700 features that are significant, then 7 would actually not be. ...
- 135
0
votes
0
answers
46
views
Harmonic mean of false positive and false discovery rates (analogous to F1)
F1 is the harmonic mean of recall (aka sensitivity, or true positive rate, TPR) and precision (aka positive predictive value, PPV).
$\text{TPR} = \text{Pr(predicted:Pos | Pos)} =$ TP/P (wikipedia ...
- 101
1
vote
1
answer
664
views
False discovery rate correction when all P value are equal
If we are have calculated multiple P values from independent tests, we can correct for the False Discovery Rate by using:
...
2
votes
1
answer
334
views
running multiple t-tests across large dataset
I'm trying to work out if a t-test is the most appropriate in this situation:
I have a data frame which looks like the one below but my data frame has aprox 37,000 rows. I'd like to run a t-test on ...
- 163
1
vote
1
answer
165
views
P-value correction for multiple Mann-Whitney tests, some of them being dependent
I have performed multiple comparisons using Mann-Whitney U tests, and want to correct the p-values to know which results are worth reporting. The structure of the data is as follows :
2 experiments, ...
- 51
0
votes
0
answers
39
views
Test for gaussian mixture fit when component assignment is known?
I have a process 𝑃 generating random variables X_1, ... X_n. From each of these I've sampled a set of samples ...
0
votes
0
answers
30
views
False Discovery Rate pvalue correction method [duplicate]
I am trying different methods to correct a p-value from simulated data,namely the following "holm", "hochberg", "hommel", "bonferroni","BH", "BY&...
- 31
1
vote
1
answer
133
views
Adjust p-value of the all output or just the variable of interest?
I am running linear mixed effect models using the lme4 package in R with the following output.
...
- 11
0
votes
0
answers
53
views
calculating expected value of false discovery rate - Benjamini-hochberg
I recently got a question that cannot seem to solve, as a beginner in the statistical field.
assume we have 20 tests.
5 with the p-value of 1/330
5 with the p-value of 1/150
10 with the p-value of 1/...
1
vote
0
answers
863
views
Do the p-values produced by lmerTest::lmer need to be adjusted for multiple comparisons?
I am running a fairly simple linear mixed model with multiple observations per patient, comparing a lab value to a reference group (healthy control). Here's some fake data:
...
- 11
0
votes
0
answers
37
views
How to adjusting p-values from results of trend analysis on water quality data
I have results from Mann Kendall test for trends at 100 sample locations for 5 different analytes at each location (i.e., 500 p-values). If I had the trend results from 500 sample locations for 1 ...
- 1
3
votes
2
answers
2k
views
Q-value for FDR correction in R stats package
I used FDR correction (i.e., Benjamin-Hochberg tests) from the R 'Stats' package, but I'm confused of the q value used in the package. The notations in the 'p.adjust' function does not include the ...
- 189
1
vote
1
answer
40
views
When running BH Procedure and there are different types of statistical tests, are they all included?
Let's say I'm running 100 statistical tests. Some of them are t tests, some are ANOVAs (and for each ANOVA subsequent post-hoc t tests should a significant result be obtained), some of them ...
1
vote
1
answer
241
views
FDR correction stronger than Bonferroni?
Is it possible to get a situation where, in a multiple-testing scenario, for some individual tests, the Benjamini&Yukatieli (2001)-correction ends up being more stringent than Bonferroni? I.e. ...
- 281
0
votes
0
answers
421
views
Should I perform multiple testing correction (FDR) for all the P-values regardless if they derive from Ttest or Wilcoxon?
There are 100 participants and I want to test if 50 metabolites change before and after intervention (antiobesity drug).
First, I check the normality for each variable, and then I perform a T-test or ...
- 1
2
votes
0
answers
33
views
False rejection control techniques review
As far as I know, in multiple hypothesis testing scenarios there are methods to control for the False Discovery Rate (FDR) and the Family-Wise Error Rate (FWER) but I'm not aware of any method for ...
- 306
0
votes
0
answers
184
views
Question about power of FDR and FWE
I understand that false discovery rate (FDR) is $\leq$ familywise error rate (FWE). I have read this means that controlling FDR is hence "more powerful" than controlling FWE. What does ...
- 523
3
votes
1
answer
86
views
Does a procedure controlling FDR at level $\alpha$ always reject at least as much as a procedure controlling FWE at $\alpha$?
I understand the false discovery rate (FDR) is weakly less than the familywise error rate (FWE), and FDR is thus a less stringent way to control for type 1 errors. However, will a procedure that ...
- 523
6
votes
3
answers
1k
views
How many coin flips are needed to reliably know a coin of weight w is unfair?
I want to find out how many flips I need to flip a coin to reliably know that it is an unfair coin.
The issue is that as the coin becomes closer to 50/50, the more false-negatives you will have if you ...
- 417
2
votes
1
answer
114
views
Is the FDR linear-step up procedure Benjamini and Hochberg valid for empircal p-values from non-parametric tests (e.g., bootstrap)?
I have empirical p-values from bootstrapped hypothesis tests with 10,000 resamples each. Because there are a finite number of resamples, some of the p-values are "0", i.e., < 0.0001 with ...
- 61
0
votes
0
answers
74
views
Analysis sense check: preregister planned contrasts which we wish to conduct even if associated ANOVA is nonsignificant?
I could use a sense check on my analysis plan for an experimental study which has not yet been run; I will preregister it once I am confident that it is reasonable. There are seven between-...
- 743
1
vote
0
answers
47
views
Is there a way to adjust FDR threshold using correlation across tests (aka, number of estimated independent tests, like in Bonferroni correction)
For Bonferroni correction, correlations can be used to inform correction of multiple testing using the spectral decomposition of matrices. For example, in GWAS, the threshold is established to be 5e-8 ...
- 13
0
votes
2
answers
381
views
How do I correct for multiple testing when I have multiple models?
I have 12 concentrations that I would like to compare between 2 groups, some subjects have 2 measures. I know the concentration is also depending on age and sex so I added those to the linear model ...
- 329
0
votes
0
answers
29
views
Multiple comparison when testing one planned hypothesis
I am running one logit model containing an interaction:
logit dv year contract_type yearXcontract+Z
where contract_type and year are dummies with 4 levels.
After ...
2
votes
0
answers
384
views
Multiple comparisons version of Kolmogorov-Smirnov test
Suppose $\mathbf{y}_{i} $ and $ \mathbf{x}_{i}\,$ are length-$m$ vectors and $D_{i}$ is some arbitrary distribution $(i=1,...,N)$. I would like to conduct the following hypothesis test:
$ H_{0}: \...
- 378
2
votes
0
answers
36
views
Correct to use FDR q-values for a small number of variables?
I would like to correct for multiple testing and I was considering using FDR q-values. In the original paper by Storey, they used this approach for genomewide data, where there are several thousands ...
- 151
2
votes
1
answer
37
views
Statistical significance and the two-trials rule in clinical trials
If the false discovery rate is known to be 0.40, how do we obtain the probability that two consecutive statistically significant results arising from an application of the two-trials rule are true ...
- 31
1
vote
1
answer
129
views
How to summarize multiple independent false discovery rates (FDRs or q-values)?
In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
- 31
7
votes
1
answer
479
views
Understanding the Benjamini-Hochberg method proof
I'm trying to understand the proof in Benjamini & Hochberg's 1995 paper, specifically the Lemma in the appendix, as the rest of the proof is short and follows it.
I got stuck somewhere after ...
- 3,181
1
vote
1
answer
34
views
Differential expression for various cell sub-types : is there a multiple comparisons problem?
Take the following scenario: differential analysis of genes expression between two conditions did not yield DE genes. However, the material for the analysis came from a mixture of cell types. It is ...
- 361
0
votes
1
answer
96
views
Calculate fraction of multiple tests for which the null hypothesis was rejected
Consider a test (e.g., a permutation test) that tests, at the individual level, if a binary event occured or not.
A paper that I was recently reading, for example, tested whether or not a pair of two ...
- 3
1
vote
1
answer
252
views
If one performs FDR corrections for multiple (independent) comparisons - what is the overall FDR? [duplicate]
Suppose that I perform two different comparisons on the same genomic dataset (with say 1000 genes, and three conditions).
For each comparison, I adjust the pvalues using Benjamini Hochberg, ...
- 361
6
votes
2
answers
1k
views
When to care about FDR vs when to care about FWER
When testing multiple hypotheses on the same set of collected data, we want of course to correct for the increased likelihood of false positives from multiple tests.
So far I've always used the Holm-...
- 161
1
vote
1
answer
37
views
Removal of untestable hypothesis before multiplicity correction
Dear community members,
I performed thousands of tests, each of which do some regression comparison between numbers of events in cases and controls. Each test comprised thousands of cases, a 3-fold ...
- 1,721
0
votes
0
answers
38
views
P-values from Different Distributions
I was just considering the following hypothetical scenario, where suppose we have $X_1,...,X_n$ observations, and for each observation, we then assume that each are associated respectively with the ...
- 133