# Questions tagged [familywise-error]

The tag has no usage guidance.

38 questions
Filter by
Sorted by
Tagged with
20 views

### Example of when FWER control should be preferred to FDR control

In the Benjamini & Hochberg paper (1995) which introduces the False Discovery Rate (FDR), the authors compare the FDR to the Family-Wise Error Rate (FWER): Often the control of the FWER is not ...
11 views

### Adjusting p values when doing several comparisons between nested linear models

I have a linear model (in R) like this one (all variables are continous): mod0 <- lm_robust(DV ~ IV1 + IV2 + IV3 + IV4, data = df) (DV is stage of acquisition ...
11 views

### Why do we calculate the family-wise Type-I error rate by assuming statistical independence of pair-wise comparisons?

$FWER = 1-\big(1-\alpha\big)^m$ The above formula is typically used when calculating the $FWER$ where alpha is the criterion applied by the researcher to each test and $m$ is the number of tests. ...
26 views

### Can FWER corrected p-values be called q-values?

This is a very naive question but I feel that many people in research do not have a very clear guidelines about what can or cannot be called a "q-value". I thought I had it right but I find so many ...
28 views

### How correction for p values work in case of multiple comparison?

This is a very basic question. I want to conduct post-hoc comparison using a fitted mixed model. emmeans(model, pairwise~condition) ...
8 views

### Multiple hypothesis correction yields higher FDR on lower number of tests. Why?

I have a DF (df1) with thousands of rows, where each row has a test and associated P, and their corresponding corrected P values (FDR Benjamin-Hochberg). Under ...
12 views

### Multiple Comparisons Problem for non-independant subsets

I'm investigating whether a program was effective or not using BACI analysis and noticed that overall it had no effect (The 2-way ANOVA and non-parametric approach gave a p-value of about 0.5). I want ...
29 views

### Considering pre-tests in family-wise error correction doing an ANOVA

When I do multiple tests on a dataset, I need to correct for the family-wise error (e.g. using Bonferroni method when performing multiple pairwise t-tests). If I do an ANOVA, I need to make sure that ...
170 views

### Holm-Bonferroni correction with two interactions

I try to get a sense of the correct adjusted Holm-Bonferroni significance levels $\alpha$ when dealing with multiple interaction terms. To simplify, assume one has the following model:  y=\beta_0 + \...
75 views

### Should I adjust for family-wise error when doing many logistic regressions and t tests

In my biomarker analysis, there are 100 biomarkers. Each biomarker is measured at 4 different time points. I'm looking for any biomarker that is associated with outcome variable at any timepoints. ...
165 views

### How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
3k views

### What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
54 views

### Help with power analyses - adjusting alpha level correctly

I will be conducting a series of ANCOVAs and MANCOVAs for a 2 x 3 x 3 design with 2 covariates (1 covariate will only be relevant for a subset of analyses). There are 9 main DVs in the study. Three of ...
1k views

### Multiple comparisons in mixed effects model

tl;dr In a random-slopes model, how should one adjust for multiple comparisons when performing inference on the group-specific slopes (the BLUPs)? Note 1: Bretz et al, the R package 'multcomp', and ...
469 views

### Are family-wise error rate procedures for multiple hypotheses tests broken because there is no rigorous definition of “family”?

For example, a researcher could cleverly adjust how may hypotheses tests fall within the "family" definition in order to achieve more (or fewer) rejections when calculating the Bonferroni correction. ...
1k views

### fpr, fdr and fwe for feature selection

Scikit-learn's univariate feature selection module offers three similar sounding methods for feature selection SelectFpr - false positive rate ...
321 views

### Are type I error & FWER both conditional probabilities?

According to my textbook, type I error is defined as the conditional probability of rejecting $H_0$ given $H_0$ is correct. On the other hand, when I study the FWER (although the definition is related ...
155 views

### Combining two hypothesis tests

I have data with $N$ genes, each having some $m$ observations. I am testing two hypothesis for each gene. These hypotheses are distinct (they test different parameters), but are assumed to be ...
42 views

### What multiple comparisons correction should i use?

I have a couple of situations in which I am unsure as to which statistical FWER correction to use. 1) This will lead to a paper. I have 1 group of people that was tested twice with certain questions ...
1k views

### Testing for pairwise proportion differences

I have survey data in which participants had to tell (yes / no) whether a number of reasons could account for their desire to cease smoking. For instance, fear of... ...
53 views

### how to recover power lost due to correcting for multiple comparisons?

The right hand side plot descrives 7 treatments over the same 6 animals. The groups are paired. Measurements are log2 transformed (important for the design of the study) The assumtions for repeated ...
5k views

### FWER, FDR and multiple comparisons for Beginners

Wikipedia: "In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses ...
80 views

### Permutation test with imbalanced samples

Power is a function of sample size. Significant findings are more likely if the sample size is large. With multiple comparisons, permutation testing gives an efficient significance level that ...
786 views

### Does the logic of “family-wise error” apply to effect size estimation?

Background: As I understand, family-wise error refers to the inflation of Type I error when performing multiple hypothesis tests. For example, if I were to perform multiple post-hoc comparisons ...
147 views

### Familywise error with Bayes Factor

Recently a paper indicated a method to calculate Bayes Factor for correlations (http://link.springer.com/article/10.3758/s13423-012-0295-x/fulltext.html). This method doesn't use significance testing ...
276 views

### Help understanding successive maximization in step-down maxT algorithm

I am currently trying to better understand different permutation-based procedures for controlling FWER. Specifically, the minP/maxT procedures that are commonly used. One of the problems I am running ...
577 views

### Sample Size for Multiple Testing

I want to calculate sample size for a classic multiple testing problem of multiple doses in clinical trials. I have 3 groups: "high dose", "low dose" and "placebo", and I wish to test if at least one ...
409 views

### Planned comparisons and experiment-wise error rate in factorial experiments

When using planned comparisons in the context of factorial experiments is it still necessary to compensate for the inflation of the experiment-wise error rate? My planned comparisons involve 4 ...
39 views

### Replacing FWER corrections with binomial tests

Say I have $m$ hypotheses to test and I'd like to test them using classical hypothesis tests (i.e. with $p$ values) as part of some simultaneous inference at a confidence level of $\alpha$. Rather ...
117 views

### How do I use FDR in this situation?

Firstly, I am not a statistician so apologies if attempt of a solution is ridiculous. I am trying to come up with an algorithm that estimates a vector $v$ thats contaminated with noise Suppose we ...
1k views

### Plain language meaning of “dependent” and “independent” tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...
460 views

### How is plot of p-values distribution related to calculation of the false discovery rate?

I'm running t-tests for all combinations of pairwise comparisons of average length of stay for patients among 30 hospitals. My first approach was to use a Bonferroni corrected significance level of ....
236 views

### FWER correction for paired t-test

I have a continuous variable, and a small sample of 8 observations at 4 different time points (before some intervention, and at 3 follow-up time points). My sample, being a preliminary experiment, ...
313 views

### How to describe three groups where the means of two groups are significantly different, but neither of the two groups differ from a third group?

I am comparing 3 means from normal distributions using student's test: $\mu{_1}, \mu{_2}, \mu{_3}$ Results are: $\mu{_1} = \mu{_2}, \mu{_2} = \mu{_3}, \mu{_1} \ne \mu{_3}$ If I want to group my ...
979 views

### Pairwise chi-squared tests with Bonferroni correction

I have two categorical datasets, say, $A$ and $B$, which are sparse. I would like to apply pairwise $\chi^2$ tests to a certain categories, which are sufficiently populated (say, have expected values \$...
426 views

### How do I test for homogeneity of regression in MANCOVA?

I am using a one-way independent MANCOVA with 4 dependent variables and a single covariate. My predictor is bivariate. I'm currently trying to test for heterogeneity of regression in SPSS by ...