Skip to main content

Questions tagged [false-discovery-rate]

An expected fraction of rejected null hypotheses that are falsely rejected, i.e. the fraction of significant findings that are actually not true. One method to control FDR in multiple testing is Benjamini-Hochberg procedure.

Filter by
Sorted by
Tagged with
2 votes
1 answer
51 views

What does a FDR of 0.05 really mean

I recently study about the FDR. A FDR of 0.05 indicates that around 5% of significantly reported results are actually false positives. Is this always the case? I tried this on a dataset and I found ...
Nothing's user avatar
  • 287
1 vote
1 answer
33 views

How to apply FDR correction, and which method is most appropriate?

So I am a little confused with the application of q-values or more broadly, FDR correction (BH or BKY). I have run multiple linear mixed effect models to assess the impact of medication on several ...
John's user avatar
  • 11
1 vote
0 answers
20 views

Benjamini Hochberg Procedure [closed]

I am working on a problem for class related to multiple testing where I would like to run the BH procedure with a known $\pi_{0}$, denoting the proportion of hypothesis that are truly null, given ...
Harry Lofi's user avatar
0 votes
0 answers
23 views

FDR adjustment for confidence intervals

I have a multiple regression model in R, and I'm supposed to report the beta estimates, 95 CIs, and p-values. There are a lot of variables in the model so I used the Benjamini-Hochberg method to ...
ln 1214's user avatar
1 vote
0 answers
36 views

Knockoff R package removing ALL variables based on LASSO, Why?

I am simulating a data set from a linear regression model and selecting the variables using LASSO (glmnet). The selection works relatively well with ...
Jack's user avatar
  • 11
0 votes
0 answers
27 views

False Discovery Rate-Adjusted Multiple Confidence Intervals after p-value-adjustment

I apologize in advance if my question is incomprehensible, I am very new to this topic. My data consists of 6 dependent variables (DV1-6) and 5 independent variables (IV1-5) of interest. For reasons ...
Joe55711's user avatar
1 vote
1 answer
100 views

Estimating effect after Benjamini-Hochberg procedure

I run an AB test and I have results at city level (keep in mind I run a one-sided test): ...
David Masip's user avatar
2 votes
1 answer
62 views

Quantitatively distinguishing between multiple-comparison tests

As someone relatively new to applied statistics, I have been trying to better understand some of the best practices for applying statistical methods. I have been recently trying to understand when to ...
Argon's user avatar
  • 153
0 votes
0 answers
29 views

FDR q-values interpretation

In performing 43 test on two groups of individuals (independent t-test) I obtained 27 significant results, then adjusting the pvalues with BH procedure for controlling FDR I obtained the so-called q-...
Ed9012's user avatar
  • 391
3 votes
1 answer
244 views

FDR adjusted p-values and q-values

I am currently working on adjusting p-values in R using the False Discovery Rate (FDR) method and have encountered some confusion regarding the basic concepts of ...
Ed9012's user avatar
  • 391
1 vote
1 answer
101 views

Adjusting for multiple comparisons after Kruskal-Wallis

I am comparing biomarker levels (50 total) obtained from immunoassay in a cohort divided into 3 groups (control vs. inactive vs. active). I want to use the Kruskal-Wallis test with Dunn’s test to ...
NG0429's user avatar
  • 11
1 vote
1 answer
63 views

FDR correction - different number of p-values and models/hypotheses being tested

I have $3$ predictors: $1$ binary (prenatal alcohol exposure) $1$ categorical with $4$ categories (drinking patterns e.g., light stable, heavy reducer) $1$ continuous (total drinks consumed during ...
Emma's user avatar
  • 11
2 votes
1 answer
61 views

Dealing with many positive-depenent insiginicant results in Benjamini-Hochberg-procedure [closed]

I'm dealing with a statistical problem, where I tested a bunch of hypotheses with a very strong positive dependence within certain groups of them. Many of them didn't lead to significant results. Now, ...
mizanshu's user avatar
8 votes
3 answers
587 views

How many p-value observations do you think are required before doing FDR correction

I have two sets of correlation data and my variable of interest and I am wondering if/when o apply FDR to the obtained P-Values. On one hand, I have correlations with gene expression data, so a great ...
João Duarte's user avatar
1 vote
0 answers
15 views

Should one correct for multiple comparison for secondary outcomes in RCTs or clinical trials?

This might a simple question answered elsewhere, but it comes up often in NHST discussions and I've heard conflicting statements. Imagine a metanalysis of 50 trials on 100,000 patients total to ...
DRG's user avatar
  • 323
1 vote
0 answers
98 views

Algorithm Validation and Alternative Approaches for FDR Control in Combined Datasets

We are currently working on a project involving multiple datasets reanalyzed with the same pipeline, where all data points (patient feature) have been filtered at 1% False Discovery Rate (FDR). As ...
ypriverol's user avatar
1 vote
0 answers
15 views

Calculating FDR in batches while taking into account the total number of tests

I am running a large number (potentially hundreds of millions - billions) null hypothesis significance tests (specifically a Poisson test, but the question is general). I would like to apply to them ...
user1614062's user avatar
2 votes
1 answer
83 views

Estimate the false discovery proportion for a fixed number of discoveries

Suppose we are testing a large number ($p$) hypothesis tests. Most commonly, a correction is applied to control the false discovery rate (FDR) at a low value, say $\alpha$. But now imagine a want to ...
Knarpie's user avatar
  • 1,648
1 vote
0 answers
14 views

FDR Correction needed on a pixel-wise comparison?

I have a question, which twirls my mind but I can't find a robust basis to answer it! I have a time-frequency data (let's say 5 columns x 5 rows) for each participant in two different groups. I have ...
KhonsKhandr's user avatar
1 vote
0 answers
28 views

How to Determine the Number of Independent Tests When Considering Multiple Testing

I have a question regarding how to determine the number of independent tests when performing multiple testing corrections. Suppose my research hypothesis is whether age is related to cognition decline ...
zjppdozen's user avatar
  • 347
1 vote
0 answers
88 views

Can the critical P-value returned by the Benjamini-Krieger-Yekutieli (BKY) procedure be greater than the false discovery rate?

If I use the Benjamini-Krieger-Yekutieli (BKY) procedure for an FDR correction, is it possible for the critical P-value returned to be greater than the desired false discovery rate? I just tried ...
Joel's user avatar
  • 61
4 votes
1 answer
629 views

Pairwise comparison; Tukey HSD or FDR

I have a general question about running multiple pairwise comparisons. I did some genetic work in grad school, where we (I believe) utilized a statistical test (Student's T-test?) followed by a FDR-...
Andrew's user avatar
  • 41
1 vote
0 answers
24 views

Exploratory factor analysis (EFA) for transcriptomic data

I am doing an EFA for transcriptomic data (n=202, p=190). I did log-transformed the data because of skewness. My question is, do I have to do false discovery rate (FDR) analysis at all prior to my EFA....
Shehab's user avatar
  • 11
1 vote
0 answers
69 views

How do I use Benjamini-Hochberg procedure for post-test segmentation in A/B-tests correctly?

I am currently working on evaluating the results of an A/B-test. For this, I am also looking into some relevant post-test segments. Overall, I have no statistically significant difference between ...
Emily's user avatar
  • 11
0 votes
0 answers
37 views

Methods to control for false omission rate?

What methods are available to control for false omission rate in multiple comparisons, similar to the way Benjamini-Hochberg controls for false discovery rate? The goal is to filter out obvious ...
goweon's user avatar
  • 253
2 votes
1 answer
1k views

False discovery rate correction when all P value are equal

If we are have calculated multiple P values from independent tests, we can correct for the False Discovery Rate by using: ...
user1317221_G's user avatar
3 votes
1 answer
457 views

running multiple t-tests across large dataset

I'm trying to work out if a t-test is the most appropriate in this situation: I have a data frame which looks like the one below but my data frame has aprox 37,000 rows. I'd like to run a t-test on ...
Chip 's user avatar
  • 173
1 vote
1 answer
303 views

P-value correction for multiple Mann-Whitney tests, some of them being dependent

I have performed multiple comparisons using Mann-Whitney U tests, and want to correct the p-values to know which results are worth reporting. The structure of the data is as follows : 2 experiments, ...
alpagarou's user avatar
0 votes
0 answers
49 views

Test for gaussian mixture fit when component assignment is known?

I have a process 𝑃 generating random variables X_1, ... X_n. From each of these I've sampled a set of samples ...
Shay Palachy Affek's user avatar
0 votes
0 answers
30 views

False Discovery Rate pvalue correction method [duplicate]

I am trying different methods to correct a p-value from simulated data,namely the following "holm", "hochberg", "hommel", "bonferroni","BH", "BY&...
Tsio90's user avatar
  • 31
1 vote
1 answer
169 views

Adjust p-value of the all output or just the variable of interest?

I am running linear mixed effect models using the lme4 package in R with the following output. ...
Jess H's user avatar
  • 11
0 votes
0 answers
57 views

calculating expected value of false discovery rate - Benjamini-hochberg

I recently got a question that cannot seem to solve, as a beginner in the statistical field. assume we have 20 tests. 5 with the p-value of 1/330 5 with the p-value of 1/150 10 with the p-value of 1/...
StrugglingResearcher's user avatar
1 vote
0 answers
1k views

Do the p-values produced by lmerTest::lmer need to be adjusted for multiple comparisons?

I am running a fairly simple linear mixed model with multiple observations per patient, comparing a lab value to a reference group (healthy control). Here's some fake data: ...
HarD's user avatar
  • 217
0 votes
0 answers
44 views

How to adjusting p-values from results of trend analysis on water quality data

I have results from Mann Kendall test for trends at 100 sample locations for 5 different analytes at each location (i.e., 500 p-values). If I had the trend results from 500 sample locations for 1 ...
Jeff M.'s user avatar
3 votes
2 answers
3k views

Q-value for FDR correction in R stats package

I used FDR correction (i.e., Benjamin-Hochberg tests) from the R 'Stats' package, but I'm confused of the q value used in the package. The notations in the 'p.adjust' function does not include the ...
LCheng's user avatar
  • 189
1 vote
1 answer
44 views

When running BH Procedure and there are different types of statistical tests, are they all included?

Let's say I'm running 100 statistical tests. Some of them are t tests, some are ANOVAs (and for each ANOVA subsequent post-hoc t tests should a significant result be obtained), some of them ...
FastBallooningHead's user avatar
1 vote
1 answer
384 views

FDR correction stronger than Bonferroni?

Is it possible to get a situation where, in a multiple-testing scenario, for some individual tests, the Benjamini&Yukatieli (2001)-correction ends up being more stringent than Bonferroni? I.e. ...
Marc Vaisband's user avatar
0 votes
0 answers
562 views

Should I perform multiple testing correction (FDR) for all the P-values regardless if they derive from Ttest or Wilcoxon?

There are 100 participants and I want to test if 50 metabolites change before and after intervention (antiobesity drug). First, I check the normality for each variable, and then I perform a T-test or ...
Geli's user avatar
  • 1
2 votes
0 answers
33 views

False rejection control techniques review

As far as I know, in multiple hypothesis testing scenarios there are methods to control for the False Discovery Rate (FDR) and the Family-Wise Error Rate (FWER) but I'm not aware of any method for ...
DaSim's user avatar
  • 460
0 votes
0 answers
237 views

Question about power of FDR and FWE

I understand that false discovery rate (FDR) is $\leq$ familywise error rate (FWE). I have read this means that controlling FDR is hence "more powerful" than controlling FWE. What does ...
Golden_Ratio's user avatar
3 votes
1 answer
123 views

Does a procedure controlling FDR at level $\alpha$ always reject at least as much as a procedure controlling FWE at $\alpha$?

I understand the false discovery rate (FDR) is weakly less than the familywise error rate (FWE), and FDR is thus a less stringent way to control for type 1 errors. However, will a procedure that ...
Golden_Ratio's user avatar
9 votes
3 answers
2k views

How many coin flips are needed to reliably know a coin of weight w is unfair?

I want to find out how many flips I need to flip a coin to reliably know that it is an unfair coin. The issue is that as the coin becomes closer to 50/50, the more false-negatives you will have if you ...
Steven Sagona's user avatar
2 votes
1 answer
147 views

Is the FDR linear-step up procedure Benjamini and Hochberg valid for empircal p-values from non-parametric tests (e.g., bootstrap)?

I have empirical p-values from bootstrapped hypothesis tests with 10,000 resamples each. Because there are a finite number of resamples, some of the p-values are "0", i.e., < 0.0001 with ...
Joel's user avatar
  • 61
0 votes
0 answers
98 views

Analysis sense check: preregister planned contrasts which we wish to conduct even if associated ANOVA is nonsignificant?

I could use a sense check on my analysis plan for an experimental study which has not yet been run; I will preregister it once I am confident that it is reasonable. There are seven between-...
Gabriel's user avatar
  • 743
1 vote
0 answers
48 views

Is there a way to adjust FDR threshold using correlation across tests (aka, number of estimated independent tests, like in Bonferroni correction)

For Bonferroni correction, correlations can be used to inform correction of multiple testing using the spectral decomposition of matrices. For example, in GWAS, the threshold is established to be 5e-8 ...
Albert Ying's user avatar
0 votes
2 answers
532 views

How do I correct for multiple testing when I have multiple models?

I have 12 concentrations that I would like to compare between 2 groups, some subjects have 2 measures. I know the concentration is also depending on age and sex so I added those to the linear model ...
CST's user avatar
  • 339
0 votes
0 answers
33 views

Multiple comparison when testing one planned hypothesis

I am running one logit model containing an interaction: logit dv year contract_type yearXcontract+Z where contract_type and year are dummies with 4 levels. After ...
Lydia Palumbo's user avatar
2 votes
0 answers
455 views

Multiple comparisons version of Kolmogorov-Smirnov test

Suppose $\mathbf{y}_{i} $ and $ \mathbf{x}_{i}\,$ are length-$m$ vectors and $D_{i}$ is some arbitrary distribution $(i=1,...,N)$. I would like to conduct the following hypothesis test: $ H_{0}: \...
Mathew Carroll's user avatar
2 votes
0 answers
42 views

Correct to use FDR q-values for a small number of variables?

I would like to correct for multiple testing and I was considering using FDR q-values. In the original paper by Storey, they used this approach for genomewide data, where there are several thousands ...
Alex's user avatar
  • 181
2 votes
1 answer
39 views

Statistical significance and the two-trials rule in clinical trials

If the false discovery rate is known to be 0.40, how do we obtain the probability that two consecutive statistically significant results arising from an application of the two-trials rule are true ...
Purist's user avatar
  • 31

1
2 3 4 5 6