Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

67
votes
1answer
7k views

40,000 neuroscience papers might be wrong

I saw this article in the Economist about a seemingly devastating paper [1] casting doubt on "something like 40,000 published [fMRI] studies." The error, they say, is because of "erroneous statistical ...
65
votes
4answers
20k views

Look and you shall find (a correlation)

I have several hundred measurements. Now, I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are thousands of correlations. Among ...
49
votes
5answers
36k views

Is adjusting p-values in a multiple regression for multiple comparisons a good idea?

Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/dependent variables describing the demand (using the ...
42
votes
5answers
6k views

Why is multiple comparison a problem?

I find it hard to understand what really is the issue with multiple comparisons. With a simple analogy, it is said that a person who will make many decisions will make many mistakes. So very ...
41
votes
3answers
13k views

When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
34
votes
3answers
36k views

Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. Regression gives me the following $p$-...
34
votes
5answers
6k views

The meaning of “positive dependency” as a condition to use the usual method for FDR control

Benjamini and Hochberg developed the first (and still most widely used, I think) method for controlling the false discovery rate (FDR). I want to start with a bunch of P values, each for a different ...
30
votes
5answers
2k views

Data “exploration” vs data “snooping”/“torturing”?

Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem. On ...
29
votes
1answer
31k views

Multiple comparisons on a mixed effects model

I am trying to analyse some data using a mixed effect model. The data I collected represent the weight of some young animals of different genotype over time. I am using the approach proposed here: ...
28
votes
7answers
12k views

Variable selection procedure for binary classification

What are the variable/feature selection that you prefer for binary classification when there are many more variables/feature than observations in the learning set? The aim here is to discuss what is ...
25
votes
2answers
2k views

How to cope with exploratory data analysis and data dredging in small-sample studies?

Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited ...
24
votes
4answers
11k views

Correcting p values for multiple tests where tests are correlated (genetics)

I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The ...
24
votes
3answers
3k views

Why aren't multiple hypothesis corrections applied to all experiments since the dawn of time?

We know that we must apply Benjamini Hochberg-like corrections for multiple hypothesis testing to experiments based on a single data set, in order to control the false discovery rate, otherwise all ...
24
votes
2answers
4k views

Should we address multiple comparisons adjustments when using confidence intervals?

Suppose we have a multiple comparisons scenario like such as post hoc inference on pairwise statistics, or a multiple regression, where we are making a total of $m$ comparisons. Suppose also, that we ...
23
votes
4answers
32k views

Is there a nonparametric equivalent of Tukey HSD?

I am using JMP to examine differences in vegetation cover in growth form groups (trees, shrubs, forbs, etc.) before and after three treatments with a control. My sample size is small (n=5) and most of ...
23
votes
1answer
53k views

Comparing levels of factors after a GLM in R

Here is a little background about my situation: my data refer to the number of prey successfully eaten by a predator. As the number of prey is limited (25 available) in each trial, I had a column "...
22
votes
7answers
4k views

What's wrong with Bonferroni adjustments?

I read the following paper: Perneger (1998) What's wrong with Bonferroni adjustments. The author summarized by saying that Bonferroni adjustment have, at best, limited applications in biomedical ...
21
votes
4answers
27k views

Is Fisher's LSD as bad as they say it is?

When we perform experiments (on small sample sizes (usually the sample size per treatment group is about 7~8)) on two groups, we use a t-test to test for difference. However, when we perform an ANOVA (...
21
votes
4answers
7k views

Stouffer's Z-score method: what if we sum $z^2$ instead of $z$?

I am performing $N$ independent statistical tests with the same null hypothesis, and would like to combine the results into one $p$-value. It seems that there are two "accepted" methods: Fisher's ...
20
votes
4answers
18k views

Post-hocs for within subjects tests?

What is the preferred method for for conducting post-hocs for within subjects tests? I've seen published work where Tukey's HSD is employed but a review of Keppel and Maxwell & Delaney suggests ...
20
votes
4answers
2k views

If multiple comparisons are “planned”, do you still need to correct for multiple comparisons?

I'm reviewing a paper that has performed >15 separate 2x2 Chi Square tests. I've suggested that they need to correct for multiple comparisons, but they have replied saying that all the comparisons ...
20
votes
4answers
6k views

Why don't Bayesian methods require multiple testing corrections?

Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn't require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry About Multiple Comparisons, 2012. I don't ...
20
votes
5answers
606 views

When can you use data-based criteria to specify a regression model?

I've heard that when many regression model specifications (say, in OLS) are considered as possibilities for a dataset, this causes multiple comparison problems and the p-values and confidence ...
20
votes
3answers
43k views

How and when to use the Bonferroni adjustment

I have two questions regarding when to use a Bonferroni adjustment: Is it appropriate to use a Bonferroni adjustment in all cases of multiple testing? If one performs a test on a data set, then one ...
19
votes
3answers
3k views

Confusion with false discovery rate and multiple testing (on Colquhoun 2014)

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can ...
18
votes
1answer
20k views

Is Benjamini-Hochberg correction more conservative as the number of comparisons increases?

How conservative is Benjamini-Hochberg multiple testing correction relative to the total number of comparisons? For example, if I have a list of 18,000 features for two groups and I perform a Wilcoxon ...
18
votes
1answer
941 views

Plain language meaning of “dependent” and “independent” tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...
17
votes
1answer
5k views

Why use ANOVA at all instead of jumping straight into post-hoc or planned comparisons tests?

Looking at a between-groups ANOVA situation, what do you get by actually doing such an ANOVA test first, and second do post-hoc (Bonferroni, Šidák, etc.) or planned comparisons tests? Why not skip the ...
17
votes
2answers
1k views

Has anyone solved PTLOS exercise 4.1?

This an exercise given in Probability Theory: The Logic of Science by Edwin Jaynes, 2003. There is a partial solution here. I have worked out a more general partial solution, and was wondering if ...
17
votes
2answers
487 views

Independence of residuals in a computer-based experiment/simulation?

I conducted a computer-based assessment of different methods of fitting a particular type of model used in the palaeo sciences. I had a large-ish training set and so I randomly (stratified random ...
17
votes
1answer
75k views

Bonferroni or Tukey? When does the number of comparisons become large?

Reading Field's Discovering Statistics Using SPSS (3rd Edition) I was struck by a bit about post-hoc tests in ANOVA. For those wanting to control the Type I error rate he suggests Bonferroni or Tukey ...
15
votes
5answers
26k views

Similarity measures between curves?

I would like to compute the measure of similarity between two ordered sets of points---the ones under User compared with the ones under Teacher: The points are curves in 3D space, but I was thinking ...
15
votes
1answer
2k views

Why is controlling FDR less stringent than controlling FWER?

I have read that controlling FDR is less stringent than controlling FWER, such as in Wikipedia: FDR controlling procedures exert a less stringent control over false discovery compared to familywise ...
15
votes
0answers
1k views

Why is there usually no correction for multiple comparisons in Bayesian statistics? [duplicate]

How would you explain to an only moderately statistically literate scientific reviewer that adjustment for multiple comparisons is not necessary when sampling from a Bayesian posterior?
14
votes
1answer
9k views

Machine learning classifiers big-O or complexity

To evaluate the performance a new classifier algorithm, I'm trying to compare the accuracy and the complexity (big-O in training and classifying). From Machine Learning: a review I get a complete ...
14
votes
3answers
930 views

Repeated measures over time with small $n$

I was given data to analyze for a study looking at the effects of a treatment on iron levels at four different time points (before treatment, the day treatment ended, 4 weeks after treatment, and 2-4 ...
13
votes
2answers
83k views

How to obtain the results of a Tukey HSD post-hoc test in a table showing grouped pairs?

I would love to perform a TukeyHSD post-hoc test after my two-way Anova with R, obtaining a table containing the sorted pairs grouped by significant difference. (Sorry about the wording, I'm still new ...
13
votes
4answers
21k views

Post hoc tests in ANCOVA

Question: What is a good method for conducting post hoc tests of differences between group means after adjusting for the effect of a covariate? Prototypical example: Four groups, 30 participants per ...
13
votes
2answers
1k views

Sane stepwise regression?

Suppose I want to build a binary classifier. I have several thousand features and only a few 10s of samples. From domain knowledge, I have a good reason to believe that the class label can be ...
13
votes
1answer
904 views

How much smaller can $p$ values from ANOVA's $F$-test be vs. those from multiple $t$-tests on the same data?

Intro: Having noted the attention received today by this question, "Can ANOVA be significant when none of the pairwise t-tests is?," I thought I might be able to reframe it in an interesting way that ...
13
votes
2answers
388 views

Converting a list of partial rankings into a global ranking

I'm working on something like the following problem. I have a bunch of users and N books. Each user creates an ordered ranking of all the books he's read (which is likely a subset of the N books), e.g....
12
votes
1answer
11k views

Which multiple comparison method to use for a lmer model: lsmeans or glht?

I'm analyzing a data set using a mixed effects model with one fixed effect (condition) and two random effects (participant due to the within subject design and pair). The model was generated with the <...
12
votes
3answers
5k views

How to compare median survival between groups?

I'm looking into median survival using Kaplan-Meier in different states for a type of cancer. There are quite big differences between the states. How can i compare the median survival between all the ...
12
votes
1answer
4k views

Overall type I error when repeatedly testing accumulating data

I have a question about group sequential methods. According to Wikipedia: In a randomized trial with two treatment groups, classical group sequential testing is used in the following manner: If n ...
12
votes
3answers
226 views

Researcher 1 runs 1000 regressions, researcher 2 runs only 1, both get same results — should they make different inferences?

Imagine a researcher is exploring a dataset and runs 1000 different regressions and he finds one interesting relationship among them. Now imagine another researcher with the same data runs just 1 ...
12
votes
1answer
5k views

Power of FDR vs. FWER approaches in multiple comparisons [duplicate]

Regarding multiple comparisons, could someone please explain me why the power of false discovery rate (FDR) is greater than the power of family-wise error rate (FWER)?
12
votes
2answers
1k views

Adjusting the p-value for adaptive sequential analysis (for chi square test)?

I wish to know what statistical literature is relevant for the following problem, and maybe even an idea on how to solve it. Imagine the following problem: We have 4 possible treatments for some ...
11
votes
4answers
37k views

Sidak or Bonferroni?

I am using a generalized linear model in SPSS to look at the differences in average number of caterpillars (non-normal, using Tweedie distribution) on 16 different species of plants. I want to run ...
11
votes
4answers
26k views

Hypothesis test for difference in medians among more than two samples

Question The test scores of three groups of people are saved as separate vectors in R. ...
11
votes
2answers
862 views

Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?

If a team of researchers perform multiple (hypothesis) tests on a given data set, there is a volume of literature asserting that they should use some form of correction for multiple testing (...