Questions tagged [false-discovery-rate]

An expected fraction of rejected null hypotheses that are falsely rejected, i.e. the fraction of significant findings that are actually not true. One method to control FDR in multiple testing is Benjamini-Hochberg procedure.

Filter by
Sorted by
Tagged with
1 vote
1 answer
14 views

Adjust p-value of the all output or just the variable of interest?

I am running linear mixed effect models using the lme4 package in R with the following output. ...
user avatar
  • 11
0 votes
0 answers
38 views

calculating expected value of false discovery rate - Benjamini-hochberg

I recently got a question that cannot seem to solve, as a beginner in the statistical field. assume we have 20 tests. 5 with the p-value of 1/330 5 with the p-value of 1/150 10 with the p-value of 1/...
user avatar
0 votes
0 answers
26 views

Do the p-values produced by lmerTest::lmer need to be adjusted for multiple comparisons?

I am running a fairly simple linear mixed model with multiple observations per patient, comparing a lab value to a reference group (healthy control). Here's some fake data: ...
user avatar
  • 1
0 votes
0 answers
13 views

Applying threshold defined by false discovery rate to new data

I'm thinking how to correctly apply the results of statistical analysis from one experiment to new data. Suppose I have a data of two groups, n samples/instances ...
user avatar
  • 185
0 votes
0 answers
16 views

How to adjusting p-values from results of trend analysis on water quality data

I have results from Mann Kendall test for trends at 100 sample locations for 5 different analytes at each location (i.e., 500 p-values). If I had the trend results from 500 sample locations for 1 ...
user avatar
2 votes
2 answers
103 views

Q-value for FDR correction in R stats package

I used FDR correction (i.e., Benjamin-Hochberg tests) from the R 'Stats' package, but I'm confused of the q value used in the package. The notations in the 'p.adjust' function does not include the ...
user avatar
  • 69
0 votes
0 answers
6 views

How can one control for FDR if we do not know the prevalence?

I am trying to understand the concept of FDR, and particularly "controlling" the FDR in multiple comparison testing. I undesratdn, and have used many times, the concept of α (type 1 error ...
user avatar
0 votes
1 answer
18 views

When running BH Procedure and there are different types of statistical tests, are they all included?

Let's say I'm running 100 statistical tests. Some of them are t tests, some are ANOVAs (and for each ANOVA subsequent post-hoc t tests should a significant result be obtained), some of them ...
user avatar
0 votes
0 answers
5 views

After Dunn test with 4 groups, I got 6 p-values for 6 combinations. Can I just choose 4 of 6 p-values to get lower FDRs?

After Shapiro-test(p-value<0.05) and Kruskal-Wallis null rejection, Someone has done the Dunn test with 4 groups, He got 6 p-values for 6 combinations. Can he just choose 4 of 6 p-values to get ...
user avatar
1 vote
1 answer
36 views

FDR correction stronger than Bonferroni?

Is it possible to get a situation where, in a multiple-testing scenario, for some individual tests, the Benjamini&Yukatieli (2001)-correction ends up being more stringent than Bonferroni? I.e. ...
user avatar
0 votes
0 answers
38 views

Should I perform multiple testing correction (FDR) for all the P-values regardless if they derive from Ttest or Wilcoxon?

There are 100 participants and I want to test if 50 metabolites change before and after intervention (antiobesity drug). First, I check the normality for each variable, and then I perform a T-test or ...
user avatar
  • 1
1 vote
0 answers
29 views

False rejection control techniques review

As far as I know, in multiple hypothesis testing scenarios there are methods to control for the False Discovery Rate (FDR) and the Family-Wise Error Rate (FWER) but I'm not aware of any method for ...
user avatar
  • 133
0 votes
0 answers
16 views

Why is the distribution of the FDR corrected P-values shifted so drastically towards high values?

I trying to understand why my distribution of the FDR-corrected P-values (Q-values) is shifted so drastically to the high values. In this figure, on the left is the distribution of the (raw) values ...
user avatar
0 votes
0 answers
17 views

When p equals the alpha value, should we reject H0 when correcting for multiple comparison using false discovery rate?

I am using statsmodels in Python. I found a case where a p-value corrected with false discover rate (using Benjamini & Hochberg method) was corrected to be ...
user avatar
0 votes
0 answers
37 views

Question about power of FDR and FWE

I understand that false discovery rate (FDR) is $\leq$ familywise error rate (FWE). I have read this means that controlling FDR is hence "more powerful" than controlling FWE. What does ...
user avatar
3 votes
1 answer
48 views

Does a procedure controlling FDR at level $\alpha$ always reject at least as much as a procedure controlling FWE at $\alpha$?

I understand the false discovery rate (FDR) is weakly less than the familywise error rate (FWE), and FDR is thus a less stringent way to control for type 1 errors. However, will a procedure that ...
user avatar
0 votes
0 answers
43 views

Which p-values do I use for FDR in 3 separate multiple linear regression models?

I'm trying to carry out Multiple Hypothesis Testing Correction using adjusted p-values (FDR Method) from my multiple linear regression models. I am testing 11 independent variables of interests ...
user avatar
0 votes
0 answers
16 views

False Discovery Rate From Posterior Samples of Bayesian Models

This is likely a naive question but here goes. There are a number of papers on Bayesian FDR (Efron/Efron and Tibshirani) etc. All of these models seem to assume that the score(p value or z values, or ...
user avatar
6 votes
3 answers
306 views

How many coin flips are needed to reliably know a coin of weight w is unfair?

I want to find out how many flips I need to flip a coin to reliably know that it is an unfair coin. The issue is that as the coin becomes closer to 50/50, the more false-negatives you will have if you ...
user avatar
0 votes
0 answers
20 views

FDR in multiple comparisons with the same dependant variable

Let's say I have a set of 200 models that follows the structure below, my goal being to identify the genes that are significantly related to the response variable Response ~ Gene[i] + Age + Time + ...
user avatar
0 votes
0 answers
21 views

How to compute p-values for shrinkage estimates of partial correlation?

I am using the pcor.shrink function from the corpcor R package to compute the shrinkage estimates of partial correlation ...
user avatar
0 votes
0 answers
19 views

How to control for multiple testing with multiple hypotheses and multiple exploratory analyses for each hypothesis testing

I recently conducted a 3 (between-subject: two intervention groups and one control group) x 5 (within-subject: 5 dependent variables) field experiment in which two behavior interventions (indenpendent ...
user avatar
  • 31
2 votes
1 answer
28 views

Is the FDR linear-step up procedure Benjamini and Hochberg valid for empircal p-values from non-parametric tests (e.g., bootstrap)?

I have empirical p-values from bootstrapped hypothesis tests with 10,000 resamples each. Because there are a finite number of resamples, some of the p-values are "0", i.e., < 0.0001 with ...
user avatar
  • 51
0 votes
0 answers
24 views

Analysis sense check: preregister planned contrasts which we wish to conduct even if associated ANOVA is nonsignificant?

I could use a sense check on my analysis plan for an experimental study which has not yet been run; I will preregister it once I am confident that it is reasonable. There are seven between-...
user avatar
  • 683
1 vote
0 answers
22 views

Is there a way to adjust FDR threshold using correlation across tests (aka, number of estimated independent tests, like in Bonferroni correction)

For Bonferroni correction, correlations can be used to inform correction of multiple testing using the spectral decomposition of matrices. For example, in GWAS, the threshold is established to be 5e-8 ...
user avatar
0 votes
1 answer
35 views

How do I correct for multiple testing when I have multiple models?

I have 12 concentrations that I would like to compare between 2 groups, some subjects have 2 measures. I know the concentration is also depending on age and sex so I added those to the linear model ...
user avatar
  • 169
0 votes
0 answers
25 views

Multiple comparison when testing one planned hypothesis

I am running one logit model containing an interaction: logit dv year contract_type yearXcontract+Z where contract_type and year are dummies with 4 levels. After ...
user avatar
2 votes
0 answers
76 views

Multiple comparisons version of Kolmogorov-Smirnov test

Suppose $\mathbf{y}_{i} $ and $ \mathbf{x}_{i}\,$ are length-$m$ vectors and $D_{i}$ is some arbitrary distribution $(i=1,...,N)$. I would like to conduct the following hypothesis test: $ H_{0}: \...
user avatar
0 votes
0 answers
60 views

When does a proxy/substitute variable distinguish between groups as well as the original variable?

I’m trying to determine when a proxy variable can be substituted for a real variable for the purposes of distinguishing two groups. To make this more concrete, suppose I want to determine, generally, ...
user avatar
2 votes
0 answers
29 views

Correct to use FDR q-values for a small number of variables?

I would like to correct for multiple testing and I was considering using FDR q-values. In the original paper by Storey, they used this approach for genomewide data, where there are several thousands ...
user avatar
  • 111
2 votes
1 answer
33 views

Statistical significance and the two-trials rule in clinical trials

If the false discovery rate is known to be 0.40, how do we obtain the probability that two consecutive statistically significant results arising from an application of the two-trials rule are true ...
user avatar
  • 31
0 votes
0 answers
13 views

What is cFDR & does it control the FDR like BH or IHW?

Conditional False Discovery Rate (cFDR) [1] allows to use a covariate informative of power or prior likelihood but independent of the statistic/p-values under the null to increase the number of ...
user avatar
  • 341
1 vote
1 answer
67 views

How to summarize multiple independent false discovery rates (FDRs or q-values)?

In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with ...
user avatar
7 votes
1 answer
218 views

Understanding the Benjamini-Hochberg method proof

I'm trying to understand the proof in Benjamini & Hochberg's 1995 paper, specifically the Lemma in the appendix, as the rest of the proof is short and follows it. I got stuck somewhere after ...
user avatar
1 vote
1 answer
21 views

Differential expression for various cell sub-types : is there a multiple comparisons problem?

Take the following scenario: differential analysis of genes expression between two conditions did not yield DE genes. However, the material for the analysis came from a mixture of cell types. It is ...
user avatar
  • 417
0 votes
1 answer
39 views

Calculate fraction of multiple tests for which the null hypothesis was rejected

Consider a test (e.g., a permutation test) that tests, at the individual level, if a binary event occured or not. A paper that I was recently reading, for example, tested whether or not a pair of two ...
user avatar
  • 3
1 vote
1 answer
69 views

If one performs FDR corrections for multiple (independent) comparisons - what is the overall FDR? [duplicate]

Suppose that I perform two different comparisons on the same genomic dataset (with say 1000 genes, and three conditions). For each comparison, I adjust the pvalues using Benjamini Hochberg, ...
user avatar
  • 417
3 votes
1 answer
259 views

When to care about FDR vs when to care about FWER

When testing multiple hypotheses on the same set of collected data, we want of course to correct for the increased likelihood of false positives from multiple tests. So far I've always used the Holm-...
user avatar
  • 131
0 votes
1 answer
28 views

Removal of untestable hypothesis before multiplicity correction

Dear community members, I performed thousands of tests, each of which do some regression comparison between numbers of events in cases and controls. Each test comprised thousands of cases, a 3-fold ...
user avatar
0 votes
0 answers
33 views

P-values from Different Distributions

I was just considering the following hypothetical scenario, where suppose we have $X_1,...,X_n$ observations, and for each observation, we then assume that each are associated respectively with the ...
user avatar
2 votes
1 answer
180 views

P.adjust - different values when using a list vs. individual p-values

I am having some trouble determining the cause of an issue in my R coding. I have a series of 11 p-values, which correcting using the p.adjust function in R with BH corrections. When I create a list ...
user avatar
1 vote
0 answers
31 views

Corrections for multiple hypothesis testing

Apologies if this repeats a previous query, I have read a lot of prior threads (and made myself a bit cross-eyed with the ones asking whether we should correct for every hypothesis test ever performed!...
user avatar
  • 21
1 vote
1 answer
91 views

Empirical FDR from real and null distributions -- do sample sizes need to match?

Suppose I have a set of values (correlation, in this case) that come from a real distribution, and another set of values that are from a simulated null distribution. To calculate the value that ...
user avatar
0 votes
1 answer
87 views

Should I do the multiple comparisons after the 2 between-group paired t-test?

I have some problems with my statistics. I need to help. I have about 8 different variables on 14 subjects. Each variable contains 5 regional and 1 total value. Thus, the total number of variables is ...
user avatar
0 votes
1 answer
362 views

Multiple comparison correction for confidence interval

I am trying to understand multiple-comparisons adjustment for confidence intervals. I came across the following links on the topic, but a few things are not clear to me. Stanford lecture: Section 2 ...
user avatar
  • 103
1 vote
0 answers
19 views

Problem in understanding False Discovery Rate (FDR)

I am facing a problem understanding the concept of FDR or False Discovery Rate in Multiple Hypothesis Testing. May be the question is silly, please try to bear with me. Actually, when we are talking ...
user avatar
2 votes
0 answers
64 views

Multiple Comparisons Correction for Multiple ANOVA models - Include only terms of interest?

I have a a biological dataset, in which I am fitting the following partial mixed ANOVA model (apologizes for the MATLAB syntax): ...
user avatar
  • 185
12 votes
2 answers
468 views

Proof/derivation for false discovery rate in Benjamini-Hochberg procedure

The Benjamini-Hochberg procedure is a method that corrects for multiple comparisons and has a false discovery rate (FDR) equal to $\alpha$. Or is it the family wise error rate, FWER? I am a bit ...
user avatar
7 votes
3 answers
1k views

Rejection threshold of the Benjamini-Hochberg procedure

Is there a possibility to calculate or estimate the overall rejection threshold of the Benjamini–Hochberg procedure (BH)? For the correction of the FWER using the Bonferroni method, the significance ...
user avatar
0 votes
1 answer
263 views

Interpretting and Using fdrtool?

With the help of articles like this one, I am beginning to understand false discovery rate first laid out by Benjamini and Hochberg (1995). I am also seeing the value of the "unified" ...
user avatar
  • 27

1
2 3 4 5 6