Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

3
votes
1answer
174 views

Statistical testing for cohort analysis and retention: did I do this right?

A common problem in tech-startups is cohort analysis when A/B testing new app features: within a certain time period between two different dates, two different sets of users joining between those ...
0
votes
0answers
34 views

Multiple comparison correction, when?

I am not sure about how many test I have in the following scenario: I collect 3 measures of a persons performance in a task (m1, m2, m3). I also collect data on two anatomical features (A1, A2). Then ...
0
votes
0answers
171 views

Dunn's test of multiple comparisons

From the documentation page for the R function dunnTest from the package FSA: Performs Dunn's (1964) test of multiple ...
0
votes
3answers
28 views

Too many blood tests

In medicine, lab results are usually "normal" if within two standard deviations. If I were to order 20 random tests, would it be safe to assume that one "normal" result could fall outside the "normal"...
4
votes
1answer
901 views

Pairwise comparisons with emmeans for a mixed three-way interaction in a linear mixed-effects model

I have a rookie question about emmeans in R. I fit a complex model using lmer() with the following variables: A: a binary categorical predictor, within-subject B: a binary categorical predictor, ...
1
vote
1answer
32 views

How can I run multiple tests in a valid and random fashion?

I hope this is the correct place to ask this question. I'm currently doing research where I'm comparing two groups on their EEG frequency spectra. I wasn't too sure how to approach this. Data: I ...
0
votes
0answers
24 views

Testing 3 Kruskal Wallis on 3 datasets that each has 3 treatments

I am not that experienced, but, I read a lot and seems am stuck to know if what i am doing is right or not. I have 3 datasets that each were initiated based on session duration. The 1st dataset has ...
0
votes
0answers
16 views

opposite of multiple test correction

Is there a p value correction for dealing with multiple hypothesis testing when I want to know how many tests satisfy p > alpha (ie, how many tests fail to come up with a 'significant' result) instead ...
1
vote
0answers
57 views

Kolmogorov-Smirnov test for the distribution of p-values

I have a set of independent null hypothesis significance tests and their resulting p-values and I would like to make the claim that they are not all null by looking at the distribution of independent ...
0
votes
0answers
11 views

modified False Discovery Rate

In "Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection" by Candes et al, 2016, the authors discuss two notions of false discovery rate. Throughout, $q$ is the ...
0
votes
0answers
14 views

Comparing multiple independent distributions from two data sets

I am trying to find the best way to quatitatively compare two data sets. I would like to compare the width as well as the centroid of the distributions from the two data sets. Since there is many ...
1
vote
0answers
30 views

Removing non-significant main effects / interactions

I have a hypothesis that my dependent variable (a clinical outcome) is mediated both by main effects and by interactions between them. I want particularly to demonstrate that (or measure whether) the ...
1
vote
0answers
69 views

adjusted p-values for post-hoc tests?

I'm analysing about 20 lab-results of three different groups either with ANOVA, Kruskal–Wallis test or chi-squared. I correct the resulting p-values for multiple testing using the Benjamini/Hochberg ...
2
votes
0answers
25 views

Which test to use on partially overlapping groups

I have data in 4 categories, but they are not completely independent. To explain the data: I have looked at 40 Intelligent Tutoring Systems (ITSs) and determined which approach they use (model tracing,...
0
votes
0answers
44 views

Bonferroni/Holm correction when performing chi-squared residual analysis

Let's say I am performing two t-tests and one chi-squared test for a 5*2 table (for example levels of education vs. sex). I am further interested in which categories of education differ among sexes, ...
0
votes
0answers
39 views

Disparity between ANOVA results and pairwise comparisons (both heteroscedastic-consistent)

I have a data set consisting of a numerical response variable (y), and two factors (A with 3 levels, and ...
3
votes
1answer
58 views

FDR controlling demonstration in R

I'm trying to demonstrate BH method for controlling false discovery rate in R. The problem is that I get quite unexpected results (so I assume that something is going wrong with this) ...
0
votes
0answers
33 views

Cross-validation, feature subset and multiple testing correction

I have a binary prediction problem for which I am doing a 10 times 10-fold cross validation. Imagine that I do the CV analysis multiple times, each time selecting a different non-overlapping subset ...
2
votes
1answer
313 views

Testing for significant difference between coefficients from 2 different IV regressions (Stata or in general)

I'm running two different IV models in Stata that share the same DV and controls but have different instruments for the independent variable of interest. ...
0
votes
1answer
465 views

What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
1
vote
1answer
19 views

What qualifies as a set of independent tests to apply p-value adjustments?

There are many questions and answers on how to apply p-value adjustments and in which cases, but I couldn't get an answer to my problem based on those, so here goes : Assume you have a dataset with 5 ...
0
votes
0answers
59 views

Tukey's alternatives!

I have 10 independent groups and I need to know which one is significantly different than others. I am planning to use One-way ANOVA then multi-comparison test. I used Tukey's test but it always ...
2
votes
0answers
131 views

GAM - Smooth comparison with factor variables

I try to compare the splines for my 6 categories. To this end, first I tried the ordered factor approach and posted the result here. This comparison is not sufficient for me since I would like to have ...
0
votes
0answers
14 views

Are non-parametric test effective when my pairwise-evaluation data is multiplied through different distributions?

So, I've designed an experiment where I compare pairs of texts that have been clustered. I want to test a few different aspects of the data so I've created 8 different distributions and selected 100 ...
1
vote
0answers
34 views

How to adjust for the number of nested model tests?

How should I adjust for the number of tests I do when using the likelihood ratio difference test in a model estimated using maximum likelihood (or some flavor of), where the null hypothesis of the ...
-1
votes
1answer
22 views

How can I check if three or more results are statistically identical?

I conducted several experiments (for the moment 3, but there will be more in the near future), where in each experiment I made several measurements. I know (from previous experience) that the result ...
3
votes
1answer
59 views

Am I Double-Dipping on my Hypothesis Test?

I am testing a large number of simultaneous hypotheses (~300), with each hypothesis test trying to determine whether a given test statistic differs significantly from zero. Am I cheating the multiple ...
0
votes
0answers
46 views

Is scalar invariance required to compare relationships between groups (but not comparing differences in means)

I am undertaking multigroup comparisons with the aim of comparing structural relationships between groups. I have established configural and metric invariance but do not have partial scalar invariance....
0
votes
0answers
26 views

Help with power analyses - adjusting alpha level correctly

I will be conducting a series of ANCOVAs and MANCOVAs for a 2 x 3 x 3 design with 2 covariates (1 covariate will only be relevant for a subset of analyses). There are 9 main DVs in the study. Three of ...
6
votes
2answers
106 views

Drawing conclusions of several inferences with the same data in one study

In a sample size of 100, we identified the existence of two attributes A and B. Our goal is to assess whether there is any ...
1
vote
1answer
27 views

How to calculate % information in longitudinal group sequential design

I'm looking at using group sequential design for a study that will treat subjects with drug or placebo and then follow them for a year. The subjects get measured every 3 months for the year under ...
1
vote
0answers
71 views

Wilcoxon Signed Rank Test for equally likely groups

I have a data set with results from 50 subjects who each tested two different types of running shoe, A and B, on four different surfaces, (i.e. each subject conducted 8 tests each). After each test ...
0
votes
0answers
102 views

Adjust p-values returned from a binomial test for multiple testing

I'm using the binomial test in R to test whether the number of observations of X are significantly higher or lower than expected. e.g.: ...
1
vote
0answers
27 views

Correction for multiple comparisons after permutation

I've trained 17 regression models using a support vector regression machine providing correlations (r) between true and predicted labels and the mean squared error (MSE) for each model. The target ...
1
vote
1answer
240 views

About Kruskal-Wallis, Mann-Whitney U and multiple comparisons correction

I have 5 independent samples which I want to compare, to define if there are significant differences in the distribution of values across samples. The samples' sizes are: 4562, 1116, 314, 151, 77 I ...
0
votes
0answers
39 views

How to visualize and represent many (>20) post-hoc hypothesis tests and p-values?

I have a repeated measure design with 2 factors, one of which has 3 levels and the other has 4 levels. This results in 66 pairwise comparisons when I use lsmeans()...
0
votes
0answers
19 views

Multiple comparisons for multiple methods on multiple independent datasets

I have some problems in mutliple comparisons. I am building a regression-based model to predict the scores of some incoming texts. In one of my experiments, I propose the hypothesis that modelling ...
0
votes
0answers
13 views

How to calculate Importance of attributes in choice task with changing attributes

I have a pairwise comparison with for 7 different items, which are compared in a pairwise comparison. The items are not directly compared, but as combinations. I.e. each test person compares a ...
0
votes
0answers
29 views

mixed / hierarchical /multilevel modelling: questions about model design

I'm new to multilevel modelling, so these are (hopefully) basic questions. :) I'll first describe the dataset and a proposed model, followed by questions of the model. The data: I have a S subjects ...
1
vote
1answer
83 views

two-way ANOVA - how to conduct post-hoc tests?

Conducted a two-way ANOVA and got a main effect of sound manipulation onto perceived diagnosticity. ...
2
votes
0answers
30 views

Compare means between months of consecutive years

I have sales data of 150 sellers of an Internet website. For each seller I have daily sales for two years, that is, 730 measures, and there are quite a few days in which the sales are 0. On the other ...
1
vote
1answer
80 views

Controlling FDR when p-values are bimodal

I have a quick question about using the Benjamini-Hochberg procedure to control FDR using the p.adjust() function in R. I have read that in order to properly apply ...
1
vote
0answers
58 views

How and when to use multiple comparison tests

There seems to be some variability in WHEN multiple comparison tests such as Bonferroni or FDR might be implemented. Type 1: One might carry out multiple one-way ANOVA's to see whether or not there ...
0
votes
0answers
28 views

P-value corrections when distribution is bimodal

I am analyzing a set of transcriptome data to assess which "GO categories" are significantly enriched using a tool called GOSeq (https://bioconductor.org/packages/release/bioc/html/goseq.html). Put ...
0
votes
1answer
41 views

How to compare several means to control when the distribution is not normal and sample sizes are unequal?

I have a large dataset (n = 170,000), distributed roughly equally among 5 groups (each is 34,000 ± ~1,000). I want to compare whether the means in any of the first 4 groups are different from the ...
1
vote
0answers
43 views

FDR and the the Benjamini-Hochberg Method

I am trying to understand the Benjamini-Hochberg Method for controlling the false discovery rate. Mathematically, if we are given with m hypothesis testing procedures, we sort the P-Values and reject ...
2
votes
1answer
803 views

Why do planned comparisons and post-hoc tests differ?

I conducted a three-way independent ANOVA. I manipulated the sound of my stimulus to be 1: congruent (KON), 2: incongruent (INK) and 3: no sound at all (control condition). One of my planned ...
0
votes
0answers
19 views

How to compare three groups of data each with a different sample size

I have a sample of 26 individuals. Those were requested to answer a questionnaire. Based on the collected info I have arranged my sample in 3 different groups: one with 16 individuals - GROUP A, ...
0
votes
0answers
42 views

Monte-Carlo Simulation Between Different Variables in the Same Model

I'd like to run a Monte Carlo simulation to test the null hypothesis that the predicted probabilities of two binary independent variables which are a part of my logistic regression model are equal ...
1
vote
0answers
14 views

How to test hypothesis with multiple 'or' restictions

I am seeking help to interpret a question and set up test procedures for a series of hypotheses (six in total, two of which I can answer). It is part of a weekly problem set we are given in class. The ...