Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Make your voice heard. Take the 2019 Developer Survey now

Questions tagged [familywise-error]

The tag has no usage guidance.

1
vote
1answer
33 views

Holm-Bonferroni correction with two interactions

I try to get a sense of the correct adjusted Holm-Bonferroni significance levels $\alpha$ when dealing with multiple interaction terms. To simplify, assume one has the following model: $$ y=\beta_0 + \...
0
votes
1answer
19 views

Should I adjust for family-wise error when doing many logistic regressions and t tests

In my biomarker analysis, there are 100 biomarkers. Each biomarker is measured at 4 different time points. I'm looking for any biomarker that is associated with outcome variable at any timepoints. ...
0
votes
0answers
36 views

Defining “family” in the Family Wise Error Rate (FWER) correction

I know that the term "family" in FWER isn't defined very clearly in the literature and although one definition that is widely accepted, "All tests that are performed", a lot of applied statisticians ...
0
votes
2answers
42 views

How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
0
votes
1answer
491 views

What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
0
votes
0answers
26 views

Help with power analyses - adjusting alpha level correctly

I will be conducting a series of ANCOVAs and MANCOVAs for a 2 x 3 x 3 design with 2 covariates (1 covariate will only be relevant for a subset of analyses). There are 9 main DVs in the study. Three of ...
3
votes
1answer
276 views

Multiple comparisons in mixed effects model

tl;dr In a random-slopes model, how should one adjust for multiple comparisons when performing inference on the group-specific slopes (the BLUPs)? Note 1: Bretz et al, the R package 'multcomp', and ...
1
vote
2answers
215 views

Are family-wise error rate procedures for multiple hypotheses tests broken because there is no rigorous definition of “family”?

For example, a researcher could cleverly adjust how may hypotheses tests fall within the "family" definition in order to achieve more (or fewer) rejections when calculating the Bonferroni correction. ...
0
votes
0answers
11 views

Dunnett's Test Without ANOVA Protection

I understand that the Dunnett's test, among others, does not require an ANOVA prior to analysis to "protect" the family-wise error rate, as the test is able to do this itself. Does anyone have some ...
0
votes
1answer
257 views

fpr, fdr and fwe for feature selection

Scikit-learn's univariate feature selection module offers three similar sounding methods for feature selection SelectFpr - false positive rate ...
2
votes
1answer
167 views

Are type I error & FWER both conditional probabilities?

According to my textbook, type I error is defined as the conditional probability of rejecting $H_0$ given $H_0$ is correct. On the other hand, when I study the FWER (although the definition is related ...
1
vote
0answers
78 views

Combining two hypothesis tests

I have data with $N$ genes, each having some $m$ observations. I am testing two hypothesis for each gene. These hypotheses are distinct (they test different parameters), but are assumed to be ...
1
vote
0answers
37 views

What multiple comparisons correction should i use?

I have a couple of situations in which I am unsure as to which statistical FWER correction to use. 1) This will lead to a paper. I have 1 group of people that was tested twice with certain questions ...
0
votes
2answers
506 views

Testing for pairwise proportion differences

I have survey data in which participants had to tell (yes / no) whether a number of reasons could account for their desire to cease smoking. For instance, fear of... ...
0
votes
0answers
49 views

how to recover power lost due to correcting for multiple comparisons?

The right hand side plot descrives 7 treatments over the same 6 animals. The groups are paired. Measurements are log2 transformed (important for the design of the study) The assumtions for repeated ...
3
votes
1answer
3k views

FWER, FDR and multiple comparisons for Beginners

Wikipedia: "In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses ...
0
votes
0answers
62 views

Permutation test with imbalanced samples

Power is a function of sample size. Significant findings are more likely if the sample size is large. With multiple comparisons, permutation testing gives an efficient significance level that ...
3
votes
4answers
622 views

Does the logic of “family-wise error” apply to effect size estimation?

Background: As I understand, family-wise error refers to the inflation of Type I error when performing multiple hypothesis tests. For example, if I were to perform multiple post-hoc comparisons ...
2
votes
1answer
77 views

Familywise error with Bayes Factor

Recently a paper indicated a method to calculate Bayes Factor for correlations (http://link.springer.com/article/10.3758/s13423-012-0295-x/fulltext.html). This method doesn't use significance testing ...
1
vote
1answer
209 views

Help understanding successive maximization in step-down maxT algorithm

I am currently trying to better understand different permutation-based procedures for controlling FWER. Specifically, the minP/maxT procedures that are commonly used. One of the problems I am running ...
1
vote
1answer
329 views

Sample Size for Multiple Testing

I want to calculate sample size for a classic multiple testing problem of multiple doses in clinical trials. I have 3 groups: "high dose", "low dose" and "placebo", and I wish to test if at least one ...
1
vote
1answer
334 views

Planned comparisons and experiment-wise error rate in factorial experiments

When using planned comparisons in the context of factorial experiments is it still necessary to compensate for the inflation of the experiment-wise error rate? My planned comparisons involve 4 ...
0
votes
0answers
35 views

Replacing FWER corrections with binomial tests

Say I have $m$ hypotheses to test and I'd like to test them using classical hypothesis tests (i.e. with $p$ values) as part of some simultaneous inference at a confidence level of $\alpha$. Rather ...
2
votes
1answer
110 views

How do I use FDR in this situation?

Firstly, I am not a statistician so apologies if attempt of a solution is ridiculous. I am trying to come up with an algorithm that estimates a vector $v$ thats contaminated with noise Suppose we ...
17
votes
1answer
900 views

Plain language meaning of “dependent” and “independent” tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...
2
votes
0answers
394 views

How is plot of p-values distribution related to calculation of the false discovery rate?

I'm running t-tests for all combinations of pairwise comparisons of average length of stay for patients among 30 hospitals. My first approach was to use a Bonferroni corrected significance level of ....
0
votes
1answer
146 views

FWER correction for paired t-test

I have a continuous variable, and a small sample of 8 observations at 4 different time points (before some intervention, and at 3 follow-up time points). My sample, being a preliminary experiment, ...
2
votes
1answer
300 views

How to describe three groups where the means of two groups are significantly different, but neither of the two groups differ from a third group?

I am comparing 3 means from normal distributions using student's test: $\mu{_1}, \mu{_2}, \mu{_3}$ Results are: $\mu{_1} = \mu{_2}, \mu{_2} = \mu{_3}, \mu{_1} \ne \mu{_3} $ If I want to group my ...
3
votes
0answers
851 views

Pairwise chi-squared tests with Bonferroni correction

I have two categorical datasets, say, $A$ and $B$, which are sparse. I would like to apply pairwise $\chi^2$ tests to a certain categories, which are sufficiently populated (say, have expected values $...
1
vote
0answers
396 views

How do I test for homogeneity of regression in MANCOVA?

I am using a one-way independent MANCOVA with 4 dependent variables and a single covariate. My predictor is bivariate. I'm currently trying to test for heterogeneity of regression in SPSS by ...
2
votes
0answers
126 views

Why does max-t methods use only the maximum of generated t-values?

I am wondering why max-t methods use the maximum of the t-values to generate the distribution. Does this not make it overly conservative compared to other methods, since the critical value will be ...
0
votes
1answer
147 views

Correction for multiple comparisons with multiple response variables

I'm trying to determine what constitutes a family of hypotheses when measuring multiple response variables. In particular I have an A/B type test where I'm comparing several marketing campaigns ...