Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
88 views

Analogue of critical difference (CD) diagram for the comparison over single dataset

does anybody know any fancy way to present the performance of multiple classifiers (in terms of AUC, for example) over just 1, single dataset? I am familiar with the critical difference diagram for ...
1
vote
1answer
61 views

Formulae for FDR values [duplicate]

Can anybody give a formula for the "FDR values"? (I write "FDR values" because strictly speaking, you don't have an FDR value for each test, but an FDR value for the whole study, but nevertheles, you ...
0
votes
0answers
50 views

30 persons, 3 related readouts

I am stuck with how to best analyze the following experimental design: We analyze 3 variables (A, B, C), that are not independent from each other: if one of them is high, the other ones are expected ...
0
votes
0answers
313 views

P-value adjustment for hierarchical / multilevel data

I'm dealing with a large data set which is hierarchical in nature, that is 1000 schools, within every school 1000 students. The dependent variable is a measure of performance. Ideally I would like to ...
1
vote
1answer
23 views

Comparing same item but before and after result

I have a questionnaire for the effectiveness of the implementation of a short course. Respondents need to answer 6 items in part B of my questionnaire with 4-point likert scale (ordinal data). The ...
1
vote
1answer
160 views

2-way ANOVA multiple comparisons question

My question is about a 2-way ANOVA for an experiment. For simplicity let's say I'm using 3 methods to stimulate cells at 3 different time points, and I want to look at a resulting output variable. ...
1
vote
0answers
8 views

Kendalls u-coefficient to calculate overall consistency of pair comparison experiments

I have 8 conditions. In a pairwise experiment, I am comparing 28 pairs: condition1 vs condition2, condition3 vs. condition 4, condition1 vs condition3, condition4 vs, condition2, etc etc. Each ...
2
votes
1answer
232 views

Bonferroni correction: how does it work?

I already read the answers here: About the Bonferroni correction, but I still don't understand the concept completely. Assume the following hypothetical case, where I perform an ANOVA test to ...
0
votes
2answers
54 views

How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
0
votes
1answer
87 views

Pairwise comparisons on mcglm joint model in r

Using the mcglm package in r, I have been able to get pairwise comparisons from single-DV models but have not figured out how to get pairwise comparisons for an IV according to both DVs in a joint ...
1
vote
1answer
96 views

Understanding Kenward-Rogers Degrees of Freedom from lsmeans()

I'm trying to analyze data using a multilevel model which predicts subject response times using the experimental group and trial validity. ...
0
votes
0answers
36 views

Correct approach to performing multiple comparisons?

Preamble I have conducted an experiment to determine whether my method is sensitive to receiving the true data or random data. In all of my experiments, I worked with a collection of 1,000 data ...
0
votes
0answers
101 views

Multivariate hypothesis testing for proportions

It's very common in website optimisation to run A/B tests for conversion rates (e.g. 'buy' button is green by default, would the conversion rate from 'visit' to 'buy' be significantly larger if we ...
1
vote
0answers
89 views

What is the difference between seemingly unrelated regression (SUR) and correcting a set of OLS results for multiple comparisons?

As I understand it, the the seemingly unrelated regressions (SUR) or seemingly unrelated regression equations (SURE) models estimate a set of Ordinary Least Squares (OLS) equations where the error ...
0
votes
0answers
178 views

Bonferroni correction for pairwise proportion comparisons for more than one outcome level

Suppose I have an event that can have 3 different outcomes. This event is recorded for 21 different factor levels for a total of 100 times, and each time an outcome happens it is counted (so I am ...
1
vote
0answers
32 views

Comparing predictiveness of alternative class definitions using linear regression

I am comparing highly-correlated ways to classify patients as sick (1) or well (0). I want to see whether the way you define sickness impacts how well sickness predicts clinical outcomes. I have ...
4
votes
1answer
72 views

When testing multiple hypotheses, what does it mean when there are not enough extremes? [closed]

Suppose you are testing a large number of hypotheses, say a million. Unlike the usual situation where you have a lot of very small p-values, in this case all of your p-values are greater than 5%. ...
1
vote
0answers
18 views

Chisquare for homogenity on subsets of samples for more detailed statements?

I have two samples of six treatment groups. The outcome is an amount of positives. The treatments in both samples are the same, the measured outcome is different but dichotomous in both samples. I ...
0
votes
0answers
24 views

Should I correct for multiple comparisons after simple main effects?

I had recently a discussion with a colleague and I can't figure out who is right. Suppose you run a study with two factors: A (level 1 and level 2) and B (Level 1, level 2 and level 3). You analyse ...
1
vote
1answer
68 views

GLMM summary and multcomp differences

Thanks for taking the time to read my question. I am using R to run a GLMM because I have taken repeated measurements on 100 individuals. The explanatory variable it is a proportion so I am using ...
2
votes
1answer
72 views

My deep learning model improved - is the difference significant?

Often deep learning/machine learning models are compared just by measuring their performance on a validation dataset - the model that performs better "wins". Within null hypothesis testing, two ...
4
votes
0answers
84 views

Do results from the lme-function require adjustment of p-values?

If I run a linear mixed model with the lme() function and get results like these (comparing the score of 4 treatment groups against a placebo group): ...
0
votes
0answers
13 views

Multiple hypothesis testing: continuous vs. discrete treatments

I'm studying the effect of rainfall, cloud cover, and temperature on mood. I also want to examine whether these weather variables interact with each other. These are three separate (continuous) ...
2
votes
0answers
21 views

Are p-value corrections needed for separate tests with various grouping variables?

I have a dataset (find an example below) in which there are many factor variables (group_a, group_b, ...
1
vote
1answer
24 views

How to test for pairwise differences between factor levels while looking at an interaction with another factor?

Let's say I have a survey with questions 1 through 10 about different characteristics of animals rated from 1 to 10 and this is repeated for 5 different animals. Suppose I collect 50 surveys so my ...
3
votes
0answers
74 views

Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...
3
votes
1answer
200 views

Statistical testing for cohort analysis and retention: did I do this right?

A common problem in tech-startups is cohort analysis when A/B testing new app features: within a certain time period between two different dates, two different sets of users joining between those ...
0
votes
0answers
35 views

Multiple comparison correction, when?

I am not sure about how many test I have in the following scenario: I collect 3 measures of a persons performance in a task (m1, m2, m3). I also collect data on two anatomical features (A1, A2). Then ...
0
votes
0answers
213 views

Dunn's test of multiple comparisons

From the documentation page for the R function dunnTest from the package FSA: Performs Dunn's (1964) test of multiple ...
0
votes
3answers
29 views

Too many blood tests

In medicine, lab results are usually "normal" if within two standard deviations. If I were to order 20 random tests, would it be safe to assume that one "normal" result could fall outside the "normal"...
4
votes
1answer
1k views

Pairwise comparisons with emmeans for a mixed three-way interaction in a linear mixed-effects model

I have a rookie question about emmeans in R. I fit a complex model using lmer() with the following variables: A: a binary categorical predictor, within-subject B: a binary categorical predictor, ...
1
vote
1answer
35 views

How can I run multiple tests in a valid and random fashion?

I hope this is the correct place to ask this question. I'm currently doing research where I'm comparing two groups on their EEG frequency spectra. I wasn't too sure how to approach this. Data: I ...
0
votes
0answers
29 views

Testing 3 Kruskal Wallis on 3 datasets that each has 3 treatments

I am not that experienced, but, I read a lot and seems am stuck to know if what i am doing is right or not. I have 3 datasets that each were initiated based on session duration. The 1st dataset has ...
0
votes
0answers
17 views

opposite of multiple test correction

Is there a p value correction for dealing with multiple hypothesis testing when I want to know how many tests satisfy p > alpha (ie, how many tests fail to come up with a 'significant' result) instead ...
1
vote
0answers
83 views

Kolmogorov-Smirnov test for the distribution of p-values

I have a set of independent null hypothesis significance tests and their resulting p-values and I would like to make the claim that they are not all null by looking at the distribution of independent ...
0
votes
0answers
17 views

modified False Discovery Rate

In "Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection" by Candes et al, 2016, the authors discuss two notions of false discovery rate. Throughout, $q$ is the ...
0
votes
0answers
18 views

Comparing multiple independent distributions from two data sets

I am trying to find the best way to quatitatively compare two data sets. I would like to compare the width as well as the centroid of the distributions from the two data sets. Since there is many ...
1
vote
0answers
32 views

Removing non-significant main effects / interactions

I have a hypothesis that my dependent variable (a clinical outcome) is mediated both by main effects and by interactions between them. I want particularly to demonstrate that (or measure whether) the ...
1
vote
0answers
96 views

adjusted p-values for post-hoc tests?

I'm analysing about 20 lab-results of three different groups either with ANOVA, Kruskal–Wallis test or chi-squared. I correct the resulting p-values for multiple testing using the Benjamini/Hochberg ...
2
votes
0answers
31 views

Which test to use on partially overlapping groups

I have data in 4 categories, but they are not completely independent. To explain the data: I have looked at 40 Intelligent Tutoring Systems (ITSs) and determined which approach they use (model tracing,...
0
votes
0answers
56 views

Bonferroni/Holm correction when performing chi-squared residual analysis

Let's say I am performing two t-tests and one chi-squared test for a 5*2 table (for example levels of education vs. sex). I am further interested in which categories of education differ among sexes, ...
0
votes
0answers
49 views

Disparity between ANOVA results and pairwise comparisons (both heteroscedastic-consistent)

I have a data set consisting of a numerical response variable (y), and two factors (A with 3 levels, and ...
3
votes
1answer
73 views

FDR controlling demonstration in R

I'm trying to demonstrate BH method for controlling false discovery rate in R. The problem is that I get quite unexpected results (so I assume that something is going wrong with this) ...
0
votes
0answers
44 views

Cross-validation, feature subset and multiple testing correction

I have a binary prediction problem for which I am doing a 10 times 10-fold cross validation. Imagine that I do the CV analysis multiple times, each time selecting a different non-overlapping subset ...
2
votes
1answer
404 views

Testing for significant difference between coefficients from 2 different IV regressions (Stata or in general)

I'm running two different IV models in Stata that share the same DV and controls but have different instruments for the independent variable of interest. ...
0
votes
1answer
719 views

What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
1
vote
1answer
20 views

What qualifies as a set of independent tests to apply p-value adjustments?

There are many questions and answers on how to apply p-value adjustments and in which cases, but I couldn't get an answer to my problem based on those, so here goes : Assume you have a dataset with 5 ...
0
votes
0answers
67 views

Tukey's alternatives!

I have 10 independent groups and I need to know which one is significantly different than others. I am planning to use One-way ANOVA then multi-comparison test. I used Tukey's test but it always ...
2
votes
0answers
179 views

GAM - Smooth comparison with factor variables

I try to compare the splines for my 6 categories. To this end, first I tried the ordered factor approach and posted the result here. This comparison is not sufficient for me since I would like to have ...
0
votes
0answers
14 views

Are non-parametric test effective when my pairwise-evaluation data is multiplied through different distributions?

So, I've designed an experiment where I compare pairs of texts that have been clustered. I want to test a few different aspects of the data so I've created 8 different distributions and selected 100 ...