Questions tagged [false-discovery-rate]

An expected fraction of rejected null hypotheses that are falsely rejected, i.e. the fraction of significant findings that are actually not true. One method to control FDR in multiple testing is Benjamini-Hochberg procedure.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

FDR correction failed, but it should be meaningful!

Two groups (disease group and healthy control group), 80 variables, do t test respectively, and then do multiple comparison FDR correction, the corrected p value does not have a significant variable (...
0 votes
0 answers
11 views

What is the probabilty of a false detection from 3 different signals?

When detecting 3 signals, each at 3 sigma, what is the likelihood that the detection is false? I know how to calculate it for 1 signal, I am just not sure about three signals together.
  • 11
1 vote
1 answer
128 views

False discovery rate correction when all P value are equal

If we are have calculated multiple P values from independent tests, we can correct for the False Discovery Rate by using: ...
1 vote
1 answer
34 views

running multiple t-tests across large dataset

I'm trying to work out if a t-test is the most appropriate in this situation: I have a data frame which looks like the one below but my data frame has aprox 37,000 rows. I'd like to run a t-test on ...
  • 153
1 vote
1 answer
52 views

P-value correction for multiple Mann-Whitney tests, some of them being dependent

I have performed multiple comparisons using Mann-Whitney U tests, and want to correct the p-values to know which results are worth reporting. The structure of the data is as follows : 2 experiments, ...
0 votes
0 answers
20 views

Test for gaussian mixture fit when component assignment is known?

I have a process 𝑃 generating random variables X_1, ... X_n. From each of these I've sampled a set of samples ...
  • 191
0 votes
0 answers
27 views

False Discovery Rate pvalue correction method [duplicate]

I am trying different methods to correct a p-value from simulated data,namely the following "holm", "hochberg", "hommel", "bonferroni","BH", "BY&...
  • 31
0 votes
0 answers
8 views

how to define the scope of a family of tests when limiting false discovery rate by applying Benjamini & Hochberg (1995)

Basically, I want to apply the procedure of Benjamini & Hochberg (1995) to limit FDR, so I need to know the number of a family of tests in the first place. And this is where I find difficult to ...
  • 1
1 vote
1 answer
34 views

Adjust p-value of the all output or just the variable of interest?

I am running linear mixed effect models using the lme4 package in R with the following output. ...
  • 11
0 votes
0 answers
43 views

calculating expected value of false discovery rate - Benjamini-hochberg

I recently got a question that cannot seem to solve, as a beginner in the statistical field. assume we have 20 tests. 5 with the p-value of 1/330 5 with the p-value of 1/150 10 with the p-value of 1/...
0 votes
0 answers
84 views

Do the p-values produced by lmerTest::lmer need to be adjusted for multiple comparisons?

I am running a fairly simple linear mixed model with multiple observations per patient, comparing a lab value to a reference group (healthy control). Here's some fake data: ...
  • 1
0 votes
0 answers
19 views

Applying threshold defined by false discovery rate to new data

I'm thinking how to correctly apply the results of statistical analysis from one experiment to new data. Suppose I have a data of two groups, n samples/instances ...
  • 185
0 votes
0 answers
24 views

How to adjusting p-values from results of trend analysis on water quality data

I have results from Mann Kendall test for trends at 100 sample locations for 5 different analytes at each location (i.e., 500 p-values). If I had the trend results from 500 sample locations for 1 ...
2 votes
2 answers
487 views

Q-value for FDR correction in R stats package

I used FDR correction (i.e., Benjamin-Hochberg tests) from the R 'Stats' package, but I'm confused of the q value used in the package. The notations in the 'p.adjust' function does not include the ...
  • 79
0 votes
0 answers
6 views

How can one control for FDR if we do not know the prevalence?

I am trying to understand the concept of FDR, and particularly "controlling" the FDR in multiple comparison testing. I undesratdn, and have used many times, the concept of α (type 1 error ...
0 votes
1 answer
21 views

When running BH Procedure and there are different types of statistical tests, are they all included?

Let's say I'm running 100 statistical tests. Some of them are t tests, some are ANOVAs (and for each ANOVA subsequent post-hoc t tests should a significant result be obtained), some of them ...
0 votes
0 answers
14 views

After Dunn test with 4 groups, I got 6 p-values for 6 combinations. Can I just choose 4 of 6 p-values to get lower FDRs?

After Shapiro-test(p-value<0.05) and Kruskal-Wallis null rejection, Someone has done the Dunn test with 4 groups, He got 6 p-values for 6 combinations. Can he just choose 4 of 6 p-values to get ...
1 vote
1 answer
70 views

FDR correction stronger than Bonferroni?

Is it possible to get a situation where, in a multiple-testing scenario, for some individual tests, the Benjamini&Yukatieli (2001)-correction ends up being more stringent than Bonferroni? I.e. ...
0 votes
0 answers
128 views

Should I perform multiple testing correction (FDR) for all the P-values regardless if they derive from Ttest or Wilcoxon?

There are 100 participants and I want to test if 50 metabolites change before and after intervention (antiobesity drug). First, I check the normality for each variable, and then I perform a T-test or ...
  • 1
2 votes
0 answers
31 views

False rejection control techniques review

As far as I know, in multiple hypothesis testing scenarios there are methods to control for the False Discovery Rate (FDR) and the Family-Wise Error Rate (FWER) but I'm not aware of any method for ...
  • 143
0 votes
0 answers
49 views

Why is the distribution of the FDR corrected P-values shifted so drastically towards high values?

I trying to understand why my distribution of the FDR-corrected P-values (Q-values) is shifted so drastically to the high values. In this figure, on the left is the distribution of the (raw) values ...
user avatar
0 votes
0 answers
34 views

When p equals the alpha value, should we reject H0 when correcting for multiple comparison using false discovery rate?

I am using statsmodels in Python. I found a case where a p-value corrected with false discover rate (using Benjamini & Hochberg method) was corrected to be ...
0 votes
0 answers
84 views

Question about power of FDR and FWE

I understand that false discovery rate (FDR) is $\leq$ familywise error rate (FWE). I have read this means that controlling FDR is hence "more powerful" than controlling FWE. What does ...
3 votes
1 answer
57 views

Does a procedure controlling FDR at level $\alpha$ always reject at least as much as a procedure controlling FWE at $\alpha$?

I understand the false discovery rate (FDR) is weakly less than the familywise error rate (FWE), and FDR is thus a less stringent way to control for type 1 errors. However, will a procedure that ...
0 votes
0 answers
83 views

Which p-values do I use for FDR in 3 separate multiple linear regression models?

I'm trying to carry out Multiple Hypothesis Testing Correction using adjusted p-values (FDR Method) from my multiple linear regression models. I am testing 11 independent variables of interests ...
0 votes
0 answers
17 views

False Discovery Rate From Posterior Samples of Bayesian Models

This is likely a naive question but here goes. There are a number of papers on Bayesian FDR (Efron/Efron and Tibshirani) etc. All of these models seem to assume that the score(p value or z values, or ...
6 votes
3 answers
467 views

How many coin flips are needed to reliably know a coin of weight w is unfair?

I want to find out how many flips I need to flip a coin to reliably know that it is an unfair coin. The issue is that as the coin becomes closer to 50/50, the more false-negatives you will have if you ...
0 votes
0 answers
23 views

FDR in multiple comparisons with the same dependant variable

Let's say I have a set of 200 models that follows the structure below, my goal being to identify the genes that are significantly related to the response variable Response ~ Gene[i] + Age + Time + ...
user avatar
2 votes
1 answer
52 views

Is the FDR linear-step up procedure Benjamini and Hochberg valid for empircal p-values from non-parametric tests (e.g., bootstrap)?

I have empirical p-values from bootstrapped hypothesis tests with 10,000 resamples each. Because there are a finite number of resamples, some of the p-values are "0", i.e., < 0.0001 with ...
  • 51
0 votes
0 answers
33 views

Analysis sense check: preregister planned contrasts which we wish to conduct even if associated ANOVA is nonsignificant?

I could use a sense check on my analysis plan for an experimental study which has not yet been run; I will preregister it once I am confident that it is reasonable. There are seven between-...
  • 723
1 vote
0 answers
31 views

Is there a way to adjust FDR threshold using correlation across tests (aka, number of estimated independent tests, like in Bonferroni correction)

For Bonferroni correction, correlations can be used to inform correction of multiple testing using the spectral decomposition of matrices. For example, in GWAS, the threshold is established to be 5e-8 ...
0 votes
1 answer
132 views

How do I correct for multiple testing when I have multiple models?

I have 12 concentrations that I would like to compare between 2 groups, some subjects have 2 measures. I know the concentration is also depending on age and sex so I added those to the linear model ...
  • 189
0 votes
0 answers
27 views

Multiple comparison when testing one planned hypothesis

I am running one logit model containing an interaction: logit dv year contract_type yearXcontract+Z where contract_type and year are dummies with 4 levels. After ...
2 votes
0 answers
167 views

Multiple comparisons version of Kolmogorov-Smirnov test

Suppose $\mathbf{y}_{i} $ and $ \mathbf{x}_{i}\,$ are length-$m$ vectors and $D_{i}$ is some arbitrary distribution $(i=1,...,N)$. I would like to conduct the following hypothesis test: $ H_{0}: \...
2 votes
0 answers
33 views

Correct to use FDR q-values for a small number of variables?

I would like to correct for multiple testing and I was considering using FDR q-values. In the original paper by Storey, they used this approach for genomewide data, where there are several thousands ...
  • 121
2 votes
1 answer
34 views

Statistical significance and the two-trials rule in clinical trials

If the false discovery rate is known to be 0.40, how do we obtain the probability that two consecutive statistically significant results arising from an application of the two-trials rule are true ...
  • 31
1 vote
1 answer
78 views

How to summarize multiple independent false discovery rates (FDRs or q-values)?

In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
7 votes
1 answer
293 views

Understanding the Benjamini-Hochberg method proof

I'm trying to understand the proof in Benjamini & Hochberg's 1995 paper, specifically the Lemma in the appendix, as the rest of the proof is short and follows it. I got stuck somewhere after ...
1 vote
1 answer
21 views

Differential expression for various cell sub-types : is there a multiple comparisons problem?

Take the following scenario: differential analysis of genes expression between two conditions did not yield DE genes. However, the material for the analysis came from a mixture of cell types. It is ...
  • 417
0 votes
1 answer
63 views

Calculate fraction of multiple tests for which the null hypothesis was rejected

Consider a test (e.g., a permutation test) that tests, at the individual level, if a binary event occured or not. A paper that I was recently reading, for example, tested whether or not a pair of two ...
  • 3
1 vote
1 answer
102 views

If one performs FDR corrections for multiple (independent) comparisons - what is the overall FDR? [duplicate]

Suppose that I perform two different comparisons on the same genomic dataset (with say 1000 genes, and three conditions). For each comparison, I adjust the pvalues using Benjamini Hochberg, ...
  • 417
4 votes
1 answer
438 views

When to care about FDR vs when to care about FWER

When testing multiple hypotheses on the same set of collected data, we want of course to correct for the increased likelihood of false positives from multiple tests. So far I've always used the Holm-...
  • 141
0 votes
1 answer
28 views

Removal of untestable hypothesis before multiplicity correction

Dear community members, I performed thousands of tests, each of which do some regression comparison between numbers of events in cases and controls. Each test comprised thousands of cases, a 3-fold ...
0 votes
0 answers
35 views

P-values from Different Distributions

I was just considering the following hypothetical scenario, where suppose we have $X_1,...,X_n$ observations, and for each observation, we then assume that each are associated respectively with the ...
2 votes
1 answer
263 views

P.adjust - different values when using a list vs. individual p-values

I am having some trouble determining the cause of an issue in my R coding. I have a series of 11 p-values, which correcting using the p.adjust function in R with BH corrections. When I create a list ...
1 vote
0 answers
33 views

Corrections for multiple hypothesis testing

Apologies if this repeats a previous query, I have read a lot of prior threads (and made myself a bit cross-eyed with the ones asking whether we should correct for every hypothesis test ever performed!...
  • 21
1 vote
1 answer
155 views

Empirical FDR from real and null distributions -- do sample sizes need to match?

Suppose I have a set of values (correlation, in this case) that come from a real distribution, and another set of values that are from a simulated null distribution. To calculate the value that ...
0 votes
1 answer
140 views

Should I do the multiple comparisons after the 2 between-group paired t-test?

I have some problems with my statistics. I need to help. I have about 8 different variables on 14 subjects. Each variable contains 5 regional and 1 total value. Thus, the total number of variables is ...
0 votes
1 answer
572 views

Multiple comparison correction for confidence interval

I am trying to understand multiple-comparisons adjustment for confidence intervals. I came across the following links on the topic, but a few things are not clear to me. Stanford lecture: Section 2 ...
  • 103
1 vote
0 answers
21 views

Problem in understanding False Discovery Rate (FDR)

I am facing a problem understanding the concept of FDR or False Discovery Rate in Multiple Hypothesis Testing. May be the question is silly, please try to bear with me. Actually, when we are talking ...

1
2 3 4 5 6