Questions tagged [familywise-error]

The tag has no usage guidance.

52 questions
Filter by
Sorted by
Tagged with
33 views

ANOVA finding a main effect prevents inflation of Type I error rate

In the (free) book Understanding Statistics and Experimental Design, section 6.3, they say '...ANOVA offers a second trick. If we rejected the null hypothesis, it is appropriate to compare pairs of ...
• 123
1 vote
21 views

Benjamini Hochberg Procedure [closed]

I am working on a problem for class related to multiple testing where I would like to run the BH procedure with a known $\pi_{0}$, denoting the proportion of hypothesis that are truly null, given ...
• 345
1 vote
68 views

Should we favor family-wise error rate control or statistical power?

To counteract the multiple comparison problem, people generally use procedures that ensure strong level alpha FWER control (e.g., Bonferroni, Holm). However, these procedures necessarilly produce a ...
• 191
64 views

Quantitatively distinguishing between multiple-comparison tests

As someone relatively new to applied statistics, I have been trying to better understand some of the best practices for applying statistical methods. I have been recently trying to understand when to ...
• 153
88 views

Which family-wise correction is needed for one-sided t-tests in a randomized controlled trial with several independent measures across groups?

I conducted a randomized controlled experiment with groups $A$, $B$, $C$, and $D$, each containing around $40$ human participants who saw one of four variants of a website. Group $A$ saw a baseline, ...
• 161
100 views

How to interpret multiple hypothesis tests where $p$-values > 0.05 but test statistic signs are all (or nearly all) negative (or positive)?

This situation often arises when I'm running tests on healthcare data for my job. Taken individually, the hypothesis test $p$-values seem to show no statistically significant relationship between $X_1$...
• 6,194
109 views

What does it mean for tests to be independent?

When reading about cumulation if type-1 Error, the sentence "for independent statistical tests" occures alot, now I was wondering what this is actually means. Since tests are also random ...
• 163
453 views

Family-wise-error, what is the family?

In the last 4 hours I have been trying to wrap my head around the Error 1 Inflation that, citing wikipedia, happens, when we perfom multiple independent tests on the same dataset. The probability that ...
• 163
465 views

Specifics of post-hoc testing following significant three-way interaction in factorial anova?

TL;DR Struggling with factorial three-way anova, in particular post-hoc tests and multiplicity control. Four continuous dependent variables. Three categorical between-subject independent variables: ...
• 23
1 vote
32 views

Can multiple regression be used to test individual hypotheses?

In the linked article here Rubin proposes that familywise error correction is not necessary when the hypotheses are either that all tests will be significant ('conjunction testing') or that a series ...
• 2,043
1 vote
112 views

Do I have a multiple comparisons scenario here?

Edited after the first 2 responses, additional content in italics I have 4 treatments and a control to encourage people to sign-up for a class. I'll be looking at sign-up and actual attendance rates, ...
33 views

False rejection control techniques review

As far as I know, in multiple hypothesis testing scenarios there are methods to control for the False Discovery Rate (FDR) and the Family-Wise Error Rate (FWER) but I'm not aware of any method for ...
• 460
244 views

Question about power of FDR and FWE

I understand that false discovery rate (FDR) is $\leq$ familywise error rate (FWE). I have read this means that controlling FDR is hence "more powerful" than controlling FWE. What does ...
• 543
127 views

Does a procedure controlling FDR at level $\alpha$ always reject at least as much as a procedure controlling FWE at $\alpha$?

I understand the false discovery rate (FDR) is weakly less than the familywise error rate (FWE), and FDR is thus a less stringent way to control for type 1 errors. However, will a procedure that ...
• 543
1 vote
395 views

How to control the family-wise error rate testing simultaneous hypothesis in two-way Anova?

I guess explaining with an example would be best. Let's say we are researching which type of fertilizer and planting density produces the greatest crop yield in a field experiment. We assign different ...
• 11
3k views

When testing multiple hypotheses on the same set of collected data, we want of course to correct for the increased likelihood of false positives from multiple tests. So far I've always used the Holm-...
• 201
319 views

Example of when FWER control should be preferred to FDR control

In the Benjamini & Hochberg paper (1995) which introduces the False Discovery Rate (FDR), the authors compare the FDR to the Family-Wise Error Rate (FWER): Often the control of the FWER is not ...
• 131
1 vote
292 views

Why do we calculate the family-wise Type-I error rate by assuming statistical independence of pair-wise comparisons?

$FWER = 1-\big(1-\alpha\big)^m$ The above formula is typically used when calculating the $FWER$ where alpha is the criterion applied by the researcher to each test and $m$ is the number of tests. ...
63 views

Can FWER corrected p-values be called q-values?

This is a very naive question but I feel that many people in research do not have a very clear guidelines about what can or cannot be called a "q-value". I thought I had it right but I find so many ...
• 31
1 vote
22 views

Multiple Comparisons Problem for non-independant subsets

I'm investigating whether a program was effective or not using BACI analysis and noticed that overall it had no effect (The 2-way ANOVA and non-parametric approach gave a p-value of about 0.5). I want ...
72 views

Considering pre-tests in family-wise error correction doing an ANOVA

When I do multiple tests on a dataset, I need to correct for the family-wise error (e.g. using Bonferroni method when performing multiple pairwise t-tests). If I do an ANOVA, I need to make sure that ...
• 41
531 views

Holm-Bonferroni correction with two interactions

I try to get a sense of the correct adjusted Holm-Bonferroni significance levels $\alpha$ when dealing with multiple interaction terms. To simplify, assume one has the following model:  y=\beta_0 + \...
• 640
1 vote
725 views

Should I adjust for family-wise error when doing many logistic regressions and t tests

In my biomarker analysis, there are 100 biomarkers. Each biomarker is measured at 4 different time points. I'm looking for any biomarker that is associated with outcome variable at any timepoints. ...
• 779
364 views

How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
• 86
15k views

What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
• 139
72 views

Help with power analyses - adjusting alpha level correctly

I will be conducting a series of ANCOVAs and MANCOVAs for a 2 x 3 x 3 design with 2 covariates (1 covariate will only be relevant for a subset of analyses). There are 9 main DVs in the study. Three of ...
2k views

Multiple comparisons in mixed effects model

tl;dr In a random-slopes model, how should one adjust for multiple comparisons when performing inference on the group-specific slopes (the BLUPs)? Note 1: Bretz et al, the R package 'multcomp', and ...
• 1,908
1k views

Are family-wise error rate procedures for multiple hypotheses tests broken because there is no rigorous definition of "family"?

For example, a researcher could cleverly adjust how may hypotheses tests fall within the "family" definition in order to achieve more (or fewer) rejections when calculating the Bonferroni correction. ...
• 6,194
3k views

fpr, fdr and fwe for feature selection

Scikit-learn's univariate feature selection module offers three similar sounding methods for feature selection SelectFpr - false positive rate ...
• 1,437
637 views

Are type I error & FWER both conditional probabilities?

According to my textbook, type I error is defined as the conditional probability of rejecting $H_0$ given $H_0$ is correct. On the other hand, when I study the FWER (although the definition is related ...
• 21
1 vote
331 views

Combining two hypothesis tests

I have data with $N$ genes, each having some $m$ observations. I am testing two hypothesis for each gene. These hypotheses are distinct (they test different parameters), but are assumed to be ...
• 2,402
1 vote
49 views

What multiple comparisons correction should i use?

I have a couple of situations in which I am unsure as to which statistical FWER correction to use. 1) This will lead to a paper. I have 1 group of people that was tested twice with certain questions ...
1 vote
3k views

Testing for pairwise proportion differences

I have survey data in which participants had to tell (yes / no) whether a number of reasons could account for their desire to cease smoking. For instance, fear of... ...
• 2,139
61 views

how to recover power lost due to correcting for multiple comparisons?

The right hand side plot descrives 7 treatments over the same 6 animals. The groups are paired. Measurements are log2 transformed (important for the design of the study) The assumtions for repeated ...
• 131
9k views

FWER, FDR and multiple comparisons for Beginners

Wikipedia: "In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses ...
• 707
188 views

Permutation test with imbalanced samples

Power is a function of sample size. Significant findings are more likely if the sample size is large. With multiple comparisons, permutation testing gives an efficient significance level that ...
• 63.8k
1k views

Does the logic of "family-wise error" apply to effect size estimation?

Background: As I understand, family-wise error refers to the inflation of Type I error when performing multiple hypothesis tests. For example, if I were to perform multiple post-hoc comparisons ...
• 193
292 views

Familywise error with Bayes Factor

Recently a paper indicated a method to calculate Bayes Factor for correlations (http://link.springer.com/article/10.3758/s13423-012-0295-x/fulltext.html). This method doesn't use significance testing ...
474 views

Help understanding successive maximization in step-down maxT algorithm

I am currently trying to better understand different permutation-based procedures for controlling FWER. Specifically, the minP/maxT procedures that are commonly used. One of the problems I am running ...
• 1,893
1 vote
1k views

Sample Size for Multiple Testing

I want to calculate sample size for a classic multiple testing problem of multiple doses in clinical trials. I have 3 groups: "high dose", "low dose" and "placebo", and I wish to test if at least one ...
• 795
1 vote
556 views

Planned comparisons and experiment-wise error rate in factorial experiments

When using planned comparisons in the context of factorial experiments is it still necessary to compensate for the inflation of the experiment-wise error rate? My planned comparisons involve 4 ...
140 views

The link between FWER and max/min statistic

I am currently looking at lecture slides about fMRI data and I have the following: Let $H_{0_i}$ be the hypothesis that there is no activation in voxel $i$ (a cube area in the brain, a 3d version of a ...
• 690
51 views

Replacing FWER corrections with binomial tests

Say I have $m$ hypotheses to test and I'd like to test them using classical hypothesis tests (i.e. with $p$ values) as part of some simultaneous inference at a confidence level of $\alpha$. Rather ...
• 830
138 views

How do I use FDR in this situation?

Firstly, I am not a statistician so apologies if attempt of a solution is ridiculous. I am trying to come up with an algorithm that estimates a vector $v$ thats contaminated with noise Suppose we ...
• 690
2k views

Plain language meaning of "dependent" and "independent" tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...
• 30.3k
480 views

How is plot of p-values distribution related to calculation of the false discovery rate?

I'm running t-tests for all combinations of pairwise comparisons of average length of stay for patients among 30 hospitals. My first approach was to use a Bonferroni corrected significance level of ....
• 6,194
1 vote
396 views

FWER correction for paired t-test

I have a continuous variable, and a small sample of 8 observations at 4 different time points (before some intervention, and at 3 follow-up time points). My sample, being a preliminary experiment, ...
375 views

How to describe three groups where the means of two groups are significantly different, but neither of the two groups differ from a third group?

I am comparing 3 means from normal distributions using student's test: $\mu{_1}, \mu{_2}, \mu{_3}$ Results are: $\mu{_1} = \mu{_2}, \mu{_2} = \mu{_3}, \mu{_1} \ne \mu{_3}$ If I want to group my ...
I have two categorical datasets, say, $A$ and $B$, which are sparse. I would like to apply pairwise $\chi^2$ tests to a certain categories, which are sufficiently populated (say, have expected values \$...