Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1,195 questions
7k views

40,000 neuroscience papers might be wrong

I saw this article in the Economist about a seemingly devastating paper  casting doubt on "something like 40,000 published [fMRI] studies." The error, they say, is because of "erroneous statistical ...
20k views

Look and you shall find (a correlation)

I have several hundred measurements. Now, I am considering utilizing some kind of software to correlate every measure with every measure. This means that there are thousands of correlations. Among ...
37k views

Is adjusting p-values in a multiple regression for multiple comparisons a good idea?

Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/dependent variables describing the demand (using the ...
6k views

Why is multiple comparison a problem?

I find it hard to understand what really is the issue with multiple comparisons. With a simple analogy, it is said that a person who will make many decisions will make many mistakes. So very ...
13k views

When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that -2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
7k views

The meaning of “positive dependency” as a condition to use the usual method for FDR control

Benjamini and Hochberg developed the first (and still most widely used, I think) method for controlling the false discovery rate (FDR). I want to start with a bunch of P values, each for a different ...
36k views

Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. Regression gives me the following $p$-...
32k views

Multiple comparisons on a mixed effects model

I am trying to analyse some data using a mixed effect model. The data I collected represent the weight of some young animals of different genotype over time. I am using the approach proposed here: ...
2k views

Data “exploration” vs data “snooping”/“torturing”?

Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem. On ...
13k views

Variable selection procedure for binary classification

What are the variable/feature selection that you prefer for binary classification when there are many more variables/feature than observations in the learning set? The aim here is to discuss what is ...
2k views

How to cope with exploratory data analysis and data dredging in small-sample studies?

Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited ...
4k views

Suppose we have a multiple comparisons scenario like such as post hoc inference on pairwise statistics, or a multiple regression, where we are making a total of $m$ comparisons. Suppose also, that we ...
12k views

Correcting p values for multiple tests where tests are correlated (genetics)

I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The ...
3k views

Why aren't multiple hypothesis corrections applied to all experiments since the dawn of time?

We know that we must apply Benjamini Hochberg-like corrections for multiple hypothesis testing to experiments based on a single data set, in order to control the false discovery rate, otherwise all ...
55k views

Comparing levels of factors after a GLM in R

Here is a little background about my situation: my data refer to the number of prey successfully eaten by a predator. As the number of prey is limited (25 available) in each trial, I had a column "...
32k views

Is there a nonparametric equivalent of Tukey HSD?

I am using JMP to examine differences in vegetation cover in growth form groups (trees, shrubs, forbs, etc.) before and after three treatments with a control. My sample size is small (n=5) and most of ...
4k views

I read the following paper: Perneger (1998) What's wrong with Bonferroni adjustments. The author summarized by saying that Bonferroni adjustment have, at best, limited applications in biomedical ...
28k views

Is Fisher's LSD as bad as they say it is?

When we perform experiments (on small sample sizes (usually the sample size per treatment group is about 7~8)) on two groups, we use a t-test to test for difference. However, when we perform an ANOVA (...
7k views

Stouffer's Z-score method: what if we sum $z^2$ instead of $z$?

I am performing $N$ independent statistical tests with the same null hypothesis, and would like to combine the results into one $p$-value. It seems that there are two "accepted" methods: Fisher's ...
44k views

How and when to use the Bonferroni adjustment

I have two questions regarding when to use a Bonferroni adjustment: Is it appropriate to use a Bonferroni adjustment in all cases of multiple testing? If one performs a test on a data set, then one ...
19k views

Post-hocs for within subjects tests?

What is the preferred method for for conducting post-hocs for within subjects tests? I've seen published work where Tukey's HSD is employed but a review of Keppel and Maxwell & Delaney suggests ...
2k views

If multiple comparisons are “planned”, do you still need to correct for multiple comparisons?

I'm reviewing a paper that has performed >15 separate 2x2 Chi Square tests. I've suggested that they need to correct for multiple comparisons, but they have replied saying that all the comparisons ...
6k views

Why don't Bayesian methods require multiple testing corrections?

Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn't require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry About Multiple Comparisons, 2012. I don't ...
608 views

When can you use data-based criteria to specify a regression model?

I've heard that when many regression model specifications (say, in OLS) are considered as possibilities for a dataset, this causes multiple comparison problems and the p-values and confidence ...
3k views

Confusion with false discovery rate and multiple testing (on Colquhoun 2014)

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can ...
21k views

Is Benjamini-Hochberg correction more conservative as the number of comparisons increases?

How conservative is Benjamini-Hochberg multiple testing correction relative to the total number of comparisons? For example, if I have a list of 18,000 features for two groups and I perform a Wilcoxon ...
971 views

Plain language meaning of “dependent” and “independent” tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...
1k views

Has anyone solved PTLOS exercise 4.1?

This an exercise given in Probability Theory: The Logic of Science by Edwin Jaynes, 2003. There is a partial solution here. I have worked out a more general partial solution, and was wondering if ...
5k views

Why use ANOVA at all instead of jumping straight into post-hoc or planned comparisons tests?

Looking at a between-groups ANOVA situation, what do you get by actually doing such an ANOVA test first, and second do post-hoc (Bonferroni, Šidák, etc.) or planned comparisons tests? Why not skip the ...
489 views

Independence of residuals in a computer-based experiment/simulation?

I conducted a computer-based assessment of different methods of fitting a particular type of model used in the palaeo sciences. I had a large-ish training set and so I randomly (stratified random ...
77k views

Bonferroni or Tukey? When does the number of comparisons become large?

Reading Field's Discovering Statistics Using SPSS (3rd Edition) I was struck by a bit about post-hoc tests in ANOVA. For those wanting to control the Type I error rate he suggests Bonferroni or Tukey ...
28k views

Similarity measures between curves?

I would like to compute the measure of similarity between two ordered sets of points---the ones under User compared with the ones under Teacher: The points are curves in 3D space, but I was thinking ...
2k views

Why is controlling FDR less stringent than controlling FWER?

I have read that controlling FDR is less stringent than controlling FWER, such as in Wikipedia: FDR controlling procedures exert a less stringent control over false discovery compared to familywise ...
1k views

Why is there usually no correction for multiple comparisons in Bayesian statistics? [duplicate]

How would you explain to an only moderately statistically literate scientific reviewer that adjustment for multiple comparisons is not necessary when sampling from a Bayesian posterior?
10k views

Machine learning classifiers big-O or complexity

To evaluate the performance a new classifier algorithm, I'm trying to compare the accuracy and the complexity (big-O in training and classifying). From Machine Learning: a review I get a complete ...
956 views

Repeated measures over time with small $n$

I was given data to analyze for a study looking at the effects of a treatment on iron levels at four different time points (before treatment, the day treatment ended, 4 weeks after treatment, and 2-4 ...
86k views

How to obtain the results of a Tukey HSD post-hoc test in a table showing grouped pairs?

I would love to perform a TukeyHSD post-hoc test after my two-way Anova with R, obtaining a table containing the sorted pairs grouped by significant difference. (Sorry about the wording, I'm still new ...
21k views

Post hoc tests in ANCOVA

Question: What is a good method for conducting post hoc tests of differences between group means after adjusting for the effect of a covariate? Prototypical example: Four groups, 30 participants per ...
12k views

Which multiple comparison method to use for a lmer model: lsmeans or glht?

I'm analyzing a data set using a mixed effects model with one fixed effect (condition) and two random effects (participant due to the within subject design and pair). The model was generated with the <...
1k views

Sane stepwise regression?

Suppose I want to build a binary classifier. I have several thousand features and only a few 10s of samples. From domain knowledge, I have a good reason to believe that the class label can be ...
928 views

How much smaller can $p$ values from ANOVA's $F$-test be vs. those from multiple $t$-tests on the same data?

Intro: Having noted the attention received today by this question, "Can ANOVA be significant when none of the pairwise t-tests is?," I thought I might be able to reframe it in an interesting way that ...
409 views

Converting a list of partial rankings into a global ranking

I'm working on something like the following problem. I have a bunch of users and N books. Each user creates an ordered ranking of all the books he's read (which is likely a subset of the N books), e.g....
39k views

Sidak or Bonferroni?

I am using a generalized linear model in SPSS to look at the differences in average number of caterpillars (non-normal, using Tweedie distribution) on 16 different species of plants. I want to run ...
27k views

Hypothesis test for difference in medians among more than two samples

Question The test scores of three groups of people are saved as separate vectors in R. ...
5k views

How to compare median survival between groups?

I'm looking into median survival using Kaplan-Meier in different states for a type of cancer. There are quite big differences between the states. How can i compare the median survival between all the ...
4k views

Overall type I error when repeatedly testing accumulating data

I have a question about group sequential methods. According to Wikipedia: In a randomized trial with two treatment groups, classical group sequential testing is used in the following manner: If n ...
232 views

Researcher 1 runs 1000 regressions, researcher 2 runs only 1, both get same results — should they make different inferences?

Imagine a researcher is exploring a dataset and runs 1000 different regressions and he finds one interesting relationship among them. Now imagine another researcher with the same data runs just 1 ...
5k views

Power of FDR vs. FWER approaches in multiple comparisons [duplicate]

Regarding multiple comparisons, could someone please explain me why the power of false discovery rate (FDR) is greater than the power of family-wise error rate (FWER)?
I have two non-parametric rank correlations matrices emp and sim (for example, based on Spearman's $\rho$ rank correlation ...