Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [familywise-error]

The tag has no usage guidance.

1
vote
1answer
52 views

Holm-Bonferroni correction with two interactions

I try to get a sense of the correct adjusted Holm-Bonferroni significance levels $\alpha$ when dealing with multiple interaction terms. To simplify, assume one has the following model: $$ y=\beta_0 + \...
0
votes
1answer
24 views

Should I adjust for family-wise error when doing many logistic regressions and t tests

In my biomarker analysis, there are 100 biomarkers. Each biomarker is measured at 4 different time points. I'm looking for any biomarker that is associated with outcome variable at any timepoints. ...
0
votes
0answers
55 views

Defining “family” in the Family Wise Error Rate (FWER) correction

I know that the term "family" in FWER isn't defined very clearly in the literature and although one definition that is widely accepted, "All tests that are performed", a lot of applied statisticians ...
0
votes
2answers
54 views

How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
0
votes
1answer
709 views

What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
0
votes
0answers
31 views

Help with power analyses - adjusting alpha level correctly

I will be conducting a series of ANCOVAs and MANCOVAs for a 2 x 3 x 3 design with 2 covariates (1 covariate will only be relevant for a subset of analyses). There are 9 main DVs in the study. Three of ...
3
votes
1answer
339 views

Multiple comparisons in mixed effects model

tl;dr In a random-slopes model, how should one adjust for multiple comparisons when performing inference on the group-specific slopes (the BLUPs)? Note 1: Bretz et al, the R package 'multcomp', and ...
2
votes
2answers
246 views

Are family-wise error rate procedures for multiple hypotheses tests broken because there is no rigorous definition of “family”?

For example, a researcher could cleverly adjust how may hypotheses tests fall within the "family" definition in order to achieve more (or fewer) rejections when calculating the Bonferroni correction. ...
0
votes
0answers
20 views

Dunnett's Test Without ANOVA Protection

I understand that the Dunnett's test, among others, does not require an ANOVA prior to analysis to "protect" the family-wise error rate, as the test is able to do this itself. Does anyone have some ...
0
votes
1answer
336 views

fpr, fdr and fwe for feature selection

Scikit-learn's univariate feature selection module offers three similar sounding methods for feature selection SelectFpr - false positive rate ...
2
votes
1answer
181 views

Are type I error & FWER both conditional probabilities?

According to my textbook, type I error is defined as the conditional probability of rejecting $H_0$ given $H_0$ is correct. On the other hand, when I study the FWER (although the definition is related ...
1
vote
0answers
86 views

Combining two hypothesis tests

I have data with $N$ genes, each having some $m$ observations. I am testing two hypothesis for each gene. These hypotheses are distinct (they test different parameters), but are assumed to be ...
1
vote
0answers
37 views

What multiple comparisons correction should i use?

I have a couple of situations in which I am unsure as to which statistical FWER correction to use. 1) This will lead to a paper. I have 1 group of people that was tested twice with certain questions ...
0
votes
2answers
558 views

Testing for pairwise proportion differences

I have survey data in which participants had to tell (yes / no) whether a number of reasons could account for their desire to cease smoking. For instance, fear of... ...
0
votes
0answers
49 views

how to recover power lost due to correcting for multiple comparisons?

The right hand side plot descrives 7 treatments over the same 6 animals. The groups are paired. Measurements are log2 transformed (important for the design of the study) The assumtions for repeated ...
3
votes
1answer
4k views

FWER, FDR and multiple comparisons for Beginners

Wikipedia: "In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses ...
0
votes
0answers
66 views

Permutation test with imbalanced samples

Power is a function of sample size. Significant findings are more likely if the sample size is large. With multiple comparisons, permutation testing gives an efficient significance level that ...
3
votes
4answers
647 views

Does the logic of “family-wise error” apply to effect size estimation?

Background: As I understand, family-wise error refers to the inflation of Type I error when performing multiple hypothesis tests. For example, if I were to perform multiple post-hoc comparisons ...
2
votes
1answer
86 views

Familywise error with Bayes Factor

Recently a paper indicated a method to calculate Bayes Factor for correlations (http://link.springer.com/article/10.3758/s13423-012-0295-x/fulltext.html). This method doesn't use significance testing ...
1
vote
1answer
212 views

Help understanding successive maximization in step-down maxT algorithm

I am currently trying to better understand different permutation-based procedures for controlling FWER. Specifically, the minP/maxT procedures that are commonly used. One of the problems I am running ...
1
vote
1answer
350 views

Sample Size for Multiple Testing

I want to calculate sample size for a classic multiple testing problem of multiple doses in clinical trials. I have 3 groups: "high dose", "low dose" and "placebo", and I wish to test if at least one ...
1
vote
1answer
342 views

Planned comparisons and experiment-wise error rate in factorial experiments

When using planned comparisons in the context of factorial experiments is it still necessary to compensate for the inflation of the experiment-wise error rate? My planned comparisons involve 4 ...
0
votes
0answers
35 views

Replacing FWER corrections with binomial tests

Say I have $m$ hypotheses to test and I'd like to test them using classical hypothesis tests (i.e. with $p$ values) as part of some simultaneous inference at a confidence level of $\alpha$. Rather ...
2
votes
1answer
110 views

How do I use FDR in this situation?

Firstly, I am not a statistician so apologies if attempt of a solution is ridiculous. I am trying to come up with an algorithm that estimates a vector $v$ thats contaminated with noise Suppose we ...
18
votes
1answer
942 views

Plain language meaning of “dependent” and “independent” tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...
2
votes
0answers
418 views

How is plot of p-values distribution related to calculation of the false discovery rate?

I'm running t-tests for all combinations of pairwise comparisons of average length of stay for patients among 30 hospitals. My first approach was to use a Bonferroni corrected significance level of ....
0
votes
1answer
152 views

FWER correction for paired t-test

I have a continuous variable, and a small sample of 8 observations at 4 different time points (before some intervention, and at 3 follow-up time points). My sample, being a preliminary experiment, ...
2
votes
1answer
301 views

How to describe three groups where the means of two groups are significantly different, but neither of the two groups differ from a third group?

I am comparing 3 means from normal distributions using student's test: $\mu{_1}, \mu{_2}, \mu{_3}$ Results are: $\mu{_1} = \mu{_2}, \mu{_2} = \mu{_3}, \mu{_1} \ne \mu{_3} $ If I want to group my ...
3
votes
0answers
873 views

Pairwise chi-squared tests with Bonferroni correction

I have two categorical datasets, say, $A$ and $B$, which are sparse. I would like to apply pairwise $\chi^2$ tests to a certain categories, which are sufficiently populated (say, have expected values $...
1
vote
0answers
399 views

How do I test for homogeneity of regression in MANCOVA?

I am using a one-way independent MANCOVA with 4 dependent variables and a single covariate. My predictor is bivariate. I'm currently trying to test for heterogeneity of regression in SPSS by ...
2
votes
0answers
134 views

Why does max-t methods use only the maximum of generated t-values?

I am wondering why max-t methods use the maximum of the t-values to generate the distribution. Does this not make it overly conservative compared to other methods, since the critical value will be ...
0
votes
1answer
151 views

Correction for multiple comparisons with multiple response variables

I'm trying to determine what constitutes a family of hypotheses when measuring multiple response variables. In particular I have an A/B type test where I'm comparing several marketing campaigns ...