# Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1,191 questions
88 views

### Analogue of critical difference (CD) diagram for the comparison over single dataset

does anybody know any fancy way to present the performance of multiple classifiers (in terms of AUC, for example) over just 1, single dataset? I am familiar with the critical difference diagram for ...
61 views

### Formulae for FDR values [duplicate]

Can anybody give a formula for the "FDR values"? (I write "FDR values" because strictly speaking, you don't have an FDR value for each test, but an FDR value for the whole study, but nevertheles, you ...
50 views

### 30 persons, 3 related readouts

I am stuck with how to best analyze the following experimental design: We analyze 3 variables (A, B, C), that are not independent from each other: if one of them is high, the other ones are expected ...
313 views

### P-value adjustment for hierarchical / multilevel data

I'm dealing with a large data set which is hierarchical in nature, that is 1000 schools, within every school 1000 students. The dependent variable is a measure of performance. Ideally I would like to ...
23 views

### Comparing same item but before and after result

I have a questionnaire for the effectiveness of the implementation of a short course. Respondents need to answer 6 items in part B of my questionnaire with 4-point likert scale (ordinal data). The ...
160 views

### 2-way ANOVA multiple comparisons question

My question is about a 2-way ANOVA for an experiment. For simplicity let's say I'm using 3 methods to stimulate cells at 3 different time points, and I want to look at a resulting output variable. ...
8 views

### Kendalls u-coefficient to calculate overall consistency of pair comparison experiments

I have 8 conditions. In a pairwise experiment, I am comparing 28 pairs: condition1 vs condition2, condition3 vs. condition 4, condition1 vs condition3, condition4 vs, condition2, etc etc. Each ...
232 views

### Bonferroni correction: how does it work?

I already read the answers here: About the Bonferroni correction, but I still don't understand the concept completely. Assume the following hypothetical case, where I perform an ANOVA test to ...
54 views

### How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
87 views

### Pairwise comparisons on mcglm joint model in r

Using the mcglm package in r, I have been able to get pairwise comparisons from single-DV models but have not figured out how to get pairwise comparisons for an IV according to both DVs in a joint ...
96 views

### Understanding Kenward-Rogers Degrees of Freedom from lsmeans()

I'm trying to analyze data using a multilevel model which predicts subject response times using the experimental group and trial validity. ...
36 views

### Correct approach to performing multiple comparisons?

Preamble I have conducted an experiment to determine whether my method is sensitive to receiving the true data or random data. In all of my experiments, I worked with a collection of 1,000 data ...
101 views

### Multivariate hypothesis testing for proportions

It's very common in website optimisation to run A/B tests for conversion rates (e.g. 'buy' button is green by default, would the conversion rate from 'visit' to 'buy' be significantly larger if we ...
89 views

### What is the difference between seemingly unrelated regression (SUR) and correcting a set of OLS results for multiple comparisons?

As I understand it, the the seemingly unrelated regressions (SUR) or seemingly unrelated regression equations (SURE) models estimate a set of Ordinary Least Squares (OLS) equations where the error ...
178 views

### Bonferroni correction for pairwise proportion comparisons for more than one outcome level

Suppose I have an event that can have 3 different outcomes. This event is recorded for 21 different factor levels for a total of 100 times, and each time an outcome happens it is counted (so I am ...
32 views

### Comparing predictiveness of alternative class definitions using linear regression

I am comparing highly-correlated ways to classify patients as sick (1) or well (0). I want to see whether the way you define sickness impacts how well sickness predicts clinical outcomes. I have ...
72 views

### When testing multiple hypotheses, what does it mean when there are not enough extremes? [closed]

Suppose you are testing a large number of hypotheses, say a million. Unlike the usual situation where you have a lot of very small p-values, in this case all of your p-values are greater than 5%. ...
18 views

### Chisquare for homogenity on subsets of samples for more detailed statements?

I have two samples of six treatment groups. The outcome is an amount of positives. The treatments in both samples are the same, the measured outcome is different but dichotomous in both samples. I ...
24 views

### Should I correct for multiple comparisons after simple main effects?

I had recently a discussion with a colleague and I can't figure out who is right. Suppose you run a study with two factors: A (level 1 and level 2) and B (Level 1, level 2 and level 3). You analyse ...
68 views

### GLMM summary and multcomp differences

Thanks for taking the time to read my question. I am using R to run a GLMM because I have taken repeated measurements on 100 individuals. The explanatory variable it is a proportion so I am using ...
72 views

### My deep learning model improved - is the difference significant?

Often deep learning/machine learning models are compared just by measuring their performance on a validation dataset - the model that performs better "wins". Within null hypothesis testing, two ...
84 views

### Do results from the lme-function require adjustment of p-values?

If I run a linear mixed model with the lme() function and get results like these (comparing the score of 4 treatment groups against a placebo group): ...
13 views

### Multiple hypothesis testing: continuous vs. discrete treatments

I'm studying the effect of rainfall, cloud cover, and temperature on mood. I also want to examine whether these weather variables interact with each other. These are three separate (continuous) ...
21 views

### Are p-value corrections needed for separate tests with various grouping variables?

I have a dataset (find an example below) in which there are many factor variables (group_a, group_b, ...
24 views

### How to test for pairwise differences between factor levels while looking at an interaction with another factor?

Let's say I have a survey with questions 1 through 10 about different characteristics of animals rated from 1 to 10 and this is repeated for 5 different animals. Suppose I collect 50 surveys so my ...
74 views

### Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...
200 views

### Statistical testing for cohort analysis and retention: did I do this right?

A common problem in tech-startups is cohort analysis when A/B testing new app features: within a certain time period between two different dates, two different sets of users joining between those ...
35 views

### Multiple comparison correction, when?

I am not sure about how many test I have in the following scenario: I collect 3 measures of a persons performance in a task (m1, m2, m3). I also collect data on two anatomical features (A1, A2). Then ...
213 views

### Dunn's test of multiple comparisons

From the documentation page for the R function dunnTest from the package FSA: Performs Dunn's (1964) test of multiple ...
29 views

### Too many blood tests

In medicine, lab results are usually "normal" if within two standard deviations. If I were to order 20 random tests, would it be safe to assume that one "normal" result could fall outside the "normal"...
1k views

### Pairwise comparisons with emmeans for a mixed three-way interaction in a linear mixed-effects model

I have a rookie question about emmeans in R. I fit a complex model using lmer() with the following variables: A: a binary categorical predictor, within-subject B: a binary categorical predictor, ...
35 views

### How can I run multiple tests in a valid and random fashion?

I hope this is the correct place to ask this question. I'm currently doing research where I'm comparing two groups on their EEG frequency spectra. I wasn't too sure how to approach this. Data: I ...
29 views

### Testing 3 Kruskal Wallis on 3 datasets that each has 3 treatments

I am not that experienced, but, I read a lot and seems am stuck to know if what i am doing is right or not. I have 3 datasets that each were initiated based on session duration. The 1st dataset has ...
17 views

### opposite of multiple test correction

Is there a p value correction for dealing with multiple hypothesis testing when I want to know how many tests satisfy p > alpha (ie, how many tests fail to come up with a 'significant' result) instead ...
83 views

### Kolmogorov-Smirnov test for the distribution of p-values

I have a set of independent null hypothesis significance tests and their resulting p-values and I would like to make the claim that they are not all null by looking at the distribution of independent ...
17 views

### modified False Discovery Rate

In "Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection" by Candes et al, 2016, the authors discuss two notions of false discovery rate. Throughout, $q$ is the ...
18 views

### Comparing multiple independent distributions from two data sets

I am trying to find the best way to quatitatively compare two data sets. I would like to compare the width as well as the centroid of the distributions from the two data sets. Since there is many ...
32 views

### Removing non-significant main effects / interactions

I have a hypothesis that my dependent variable (a clinical outcome) is mediated both by main effects and by interactions between them. I want particularly to demonstrate that (or measure whether) the ...
96 views

### adjusted p-values for post-hoc tests?

I'm analysing about 20 lab-results of three different groups either with ANOVA, Kruskal–Wallis test or chi-squared. I correct the resulting p-values for multiple testing using the Benjamini/Hochberg ...
31 views

### Which test to use on partially overlapping groups

I have data in 4 categories, but they are not completely independent. To explain the data: I have looked at 40 Intelligent Tutoring Systems (ITSs) and determined which approach they use (model tracing,...
56 views

### Bonferroni/Holm correction when performing chi-squared residual analysis

Let's say I am performing two t-tests and one chi-squared test for a 5*2 table (for example levels of education vs. sex). I am further interested in which categories of education differ among sexes, ...
49 views

### Disparity between ANOVA results and pairwise comparisons (both heteroscedastic-consistent)

I have a data set consisting of a numerical response variable (y), and two factors (A with 3 levels, and ...
73 views

### FDR controlling demonstration in R

I'm trying to demonstrate BH method for controlling false discovery rate in R. The problem is that I get quite unexpected results (so I assume that something is going wrong with this) ...
44 views

### Cross-validation, feature subset and multiple testing correction

I have a binary prediction problem for which I am doing a 10 times 10-fold cross validation. Imagine that I do the CV analysis multiple times, each time selecting a different non-overlapping subset ...
404 views

### Testing for significant difference between coefficients from 2 different IV regressions (Stata or in general)

I'm running two different IV models in Stata that share the same DV and controls but have different instruments for the independent variable of interest. ...
719 views

### What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
20 views

### What qualifies as a set of independent tests to apply p-value adjustments?

There are many questions and answers on how to apply p-value adjustments and in which cases, but I couldn't get an answer to my problem based on those, so here goes : Assume you have a dataset with 5 ...
67 views

### Tukey's alternatives!

I have 10 independent groups and I need to know which one is significantly different than others. I am planning to use One-way ANOVA then multi-comparison test. I used Tukey's test but it always ...