Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

34
votes
3answers
36k views

Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. Regression gives me the following $p$-...
65
votes
4answers
20k views

Look and you shall find (a correlation)

I have several hundred measurements. Now, I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are thousands of correlations. Among ...
11
votes
2answers
864 views

Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?

If a team of researchers perform multiple (hypothesis) tests on a given data set, there is a volume of literature asserting that they should use some form of correction for multiple testing (...
49
votes
5answers
36k views

Is adjusting p-values in a multiple regression for multiple comparisons a good idea?

Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/dependent variables describing the demand (using the ...
20
votes
4answers
18k views

Post-hocs for within subjects tests?

What is the preferred method for for conducting post-hocs for within subjects tests? I've seen published work where Tukey's HSD is employed but a review of Keppel and Maxwell & Delaney suggests ...
7
votes
1answer
2k views

If a factor variable is to be dropped in model selection, should all levels be dropped simultaneously? If so, why? [duplicate]

In answer to a previous question factor pooling in model selection was discussed. If a factor or categorical variable is to be dropped in model selection, should all levels be dropped simultaneously? ...
41
votes
3answers
13k views

When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
8
votes
3answers
9k views

Relation between omnibus test and multiple comparison?

Wikipedia says Methods which rely on an omnibus test before proceeding to multiple comparisons. Typically these methods require a significant ANOVA/Tukey's range test before proceeding to ...
34
votes
5answers
6k views

The meaning of “positive dependency” as a condition to use the usual method for FDR control

Benjamini and Hochberg developed the first (and still most widely used, I think) method for controlling the false discovery rate (FDR). I want to start with a bunch of P values, each for a different ...
23
votes
1answer
53k views

Comparing levels of factors after a GLM in R

Here is a little background about my situation: my data refer to the number of prey successfully eaten by a predator. As the number of prey is limited (25 available) in each trial, I had a column "...
20
votes
4answers
6k views

Why don't Bayesian methods require multiple testing corrections?

Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn't require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry About Multiple Comparisons, 2012. I don't ...
42
votes
5answers
6k views

Why is multiple comparison a problem?

I find it hard to understand what really is the issue with multiple comparisons. With a simple analogy, it is said that a person who will make many decisions will make many mistakes. So very ...
24
votes
4answers
11k views

Correcting p values for multiple tests where tests are correlated (genetics)

I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The ...
10
votes
4answers
16k views

Multiple Chi-Squared Tests

I have cross classified data in a 2 x 2 x 6 table. Let's call the dimensions response, A and ...
29
votes
1answer
31k views

Multiple comparisons on a mixed effects model

I am trying to analyse some data using a mixed effect model. The data I collected represent the weight of some young animals of different genotype over time. I am using the approach proposed here: ...
28
votes
7answers
12k views

Variable selection procedure for binary classification

What are the variable/feature selection that you prefer for binary classification when there are many more variables/feature than observations in the learning set? The aim here is to discuss what is ...
22
votes
7answers
4k views

What's wrong with Bonferroni adjustments?

I read the following paper: Perneger (1998) What's wrong with Bonferroni adjustments. The author summarized by saying that Bonferroni adjustment have, at best, limited applications in biomedical ...
15
votes
1answer
2k views

Why is controlling FDR less stringent than controlling FWER?

I have read that controlling FDR is less stringent than controlling FWER, such as in Wikipedia: FDR controlling procedures exert a less stringent control over false discovery compared to familywise ...
10
votes
3answers
3k views

Are these formulas for transforming P, LSD, MSD, HSD, CI, to SE as an exact or inflated/conservative estimate of $\hat{\sigma}$ correct?

Background I am conducting a meta-analysis that includes previously published data. Often, differences between treatments are reported with P-values, least significant differences (LSD), and other ...
19
votes
3answers
3k views

Confusion with false discovery rate and multiple testing (on Colquhoun 2014)

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can ...
9
votes
4answers
11k views

How to carry out multiple post-hoc chi-square tests on a 2 X 3 table?

My data set is comprised of either total mortality or survival of an organism at three site types, inshore, midchannel and offshore. The numbers in the table below represent the number of sites. <...
5
votes
1answer
2k views

Linear mixed effects model and - multiplicity issue and adjusting for p-values

In our randomized controlled trial, we used linear mixed effects models to test differences between groups in changes from baseline to six months while adjusting for important covariates. We ran ...
6
votes
2answers
862 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
4
votes
4answers
2k views

About the Bonferroni correction

I have 50 variables in my dataset. I have correlated each variable against each variable, thus I have $49·50/2 = 1225$ unique correlations when no variable is tested against itself. Now, suppose I ...
25
votes
2answers
2k views

How to cope with exploratory data analysis and data dredging in small-sample studies?

Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited ...
13
votes
2answers
83k views

How to obtain the results of a Tukey HSD post-hoc test in a table showing grouped pairs?

I would love to perform a TukeyHSD post-hoc test after my two-way Anova with R, obtaining a table containing the sorted pairs grouped by significant difference. (Sorry about the wording, I'm still new ...
13
votes
4answers
21k views

Post hoc tests in ANCOVA

Question: What is a good method for conducting post hoc tests of differences between group means after adjusting for the effect of a covariate? Prototypical example: Four groups, 30 participants per ...
13
votes
1answer
905 views

How much smaller can $p$ values from ANOVA's $F$-test be vs. those from multiple $t$-tests on the same data?

Intro: Having noted the attention received today by this question, "Can ANOVA be significant when none of the pairwise t-tests is?," I thought I might be able to reframe it in an interesting way that ...
12
votes
1answer
11k views

Which multiple comparison method to use for a lmer model: lsmeans or glht?

I'm analyzing a data set using a mixed effects model with one fixed effect (condition) and two random effects (participant due to the within subject design and pair). The model was generated with the <...
10
votes
3answers
45k views

Threshold for correlation coefficient to indicate statistical significance of a correlation in a correlation matrix

I have computed a correlation matrix of a data set which contains 455 data points, each data point containing 14 characteristics. So the dimension of the correlation matrix is 14 x 14. I was ...
7
votes
3answers
12k views

Alpha adjustment for multiple testing

I understand the logic of alpha adjustment for multiple testing. However, I am confused as to whether this correction should be applied to all tests on a dataset or only the pairwise comparison in ...
7
votes
3answers
490 views

What's wrong with ''multiple testing correction'' compared to ''joint tests''?

I am wondering why it is said that multiple testing corrections are ''arbitrary'' and that they are based on a incoherent philosophy that the veracity of one statement depends on which other ...
7
votes
2answers
8k views

Has anyone used the Marascuilo procedure for comparing multiple proportions?

The Marascuilo procedure as described here seems to be a test that addresses the issue of multiple comparisons for proportions when you want to test which specific proportions are different from each ...
21
votes
4answers
27k views

Is Fisher's LSD as bad as they say it is?

When we perform experiments (on small sample sizes (usually the sample size per treatment group is about 7~8)) on two groups, we use a t-test to test for difference. However, when we perform an ANOVA (...
4
votes
2answers
16k views

How can you have a non-significant multiple regression model w/ significant predictors? [duplicate]

Possible Duplicate: Not-significant F but a significant coefficient in multiple linear regression How can a regression be significant yet all predictors be non-significant? Significance of ...
2
votes
2answers
6k views

Does one need to adjust for multiple comparisons when using MANOVA?

There are 10 dependent variables and 2 groups. When I wanted to compare 10 dependent variables between two groups, I performed MANOVA. I wonder if there is a multiple comparison problem in that case. ...
5
votes
2answers
2k views

Correlation analysis and correcting $p$-values for multiple testing

I am investigating the effect of a drug on EEG and cognition in epilepsy patients. We have done EEG and neuropsychological (NP) tests twice, before and after medical treatment. Some of parameters in ...
3
votes
2answers
10k views

Multiple t-tests vs. one-way ANOVA

I'm working on a classification problem and I have a very high F1 baseline of 85%. I have trained three classification models and I want to know which one is the best. How can I do so? I tried two ...
9
votes
1answer
509 views

What's the formula for the Benjamini-Hochberg adjusted p-value?

I understand the procedure and what it controls. So what's the formula for the adjusted p-value in the BH procedure for multiple comparisons? Just now I realized the original BH didn't produce ...
11
votes
4answers
37k views

Sidak or Bonferroni?

I am using a generalized linear model in SPSS to look at the differences in average number of caterpillars (non-normal, using Tweedie distribution) on 16 different species of plants. I want to run ...
21
votes
4answers
7k views

Stouffer's Z-score method: what if we sum $z^2$ instead of $z$?

I am performing $N$ independent statistical tests with the same null hypothesis, and would like to combine the results into one $p$-value. It seems that there are two "accepted" methods: Fisher's ...
7
votes
2answers
695 views

Is there a multiple testing problem when performing t-tests for multiple coeffcients in linear regression?

This question comes from a discussion on the recent post by @rvl It's all in the family; but do we include the in-laws too? Here's a common scenario that I've seen many times. A researcher runs a ...
20
votes
5answers
608 views

When can you use data-based criteria to specify a regression model?

I've heard that when many regression model specifications (say, in OLS) are considered as possibilities for a dataset, this causes multiple comparison problems and the p-values and confidence ...
20
votes
4answers
2k views

If multiple comparisons are “planned”, do you still need to correct for multiple comparisons?

I'm reviewing a paper that has performed >15 separate 2x2 Chi Square tests. I've suggested that they need to correct for multiple comparisons, but they have replied saying that all the comparisons ...
13
votes
2answers
1k views

Sane stepwise regression?

Suppose I want to build a binary classifier. I have several thousand features and only a few 10s of samples. From domain knowledge, I have a good reason to believe that the class label can be ...
9
votes
1answer
7k views

What are Hommel Hochberg corrections?

I have recently been introduced to to Hommel Hochberg corrections. I am trying to find a simple explanation about what this actually is/does, but am having no luck. Can anyone please give a brief and ...
9
votes
3answers
2k views

What might be a clear, practical definition for a “family of hypotheses” (with respect to familywise error rate)?

When trying to evaluate what constitutes a family of hypotheses within an experiment/project/analysis, I've found "similar in purpose" and "similar in content" given as guidelines for delimiting ...
7
votes
4answers
11k views

Friedman's test is very significant, but its post hoc comparisons (SPSS) are not significant

I ran a non-parametric Friedman's test for my data in SPSS 22 and significantly rejected the null. That would mean that among the $k$ paired samples (3 in my case), there should be detected at least ...
6
votes
4answers
2k views

If a statistics doesn't reveal a significance do I have to calculate power for it?

Following the design and data described in this question, I did a simple one-way within-subjects repeated-measures (RM) ANOVA and found some significant p-values. I then applied non-orthogonal post-...
1
vote
1answer
9k views

Two questions about Bonferroni adjustment

I have two questions about Bonferroni adjustments: 1). Can one use the Bonferroni method to compare independent groups? The reason why I ask this is it seems that many examples I've encountered ...