# Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1,191 questions
25 views

### Implications of descriptive statistics on parameter estimates

I am working on my dissertation and a question dawned on me after I ran my analyses. I studied whether the interaction between marital conflict and gratitude on mental health differed between first-...
37 views

### How to use Bonferroni correction - correct by no of parameters? No of models per dependent variable?

I have a simple question about when corrections such as Bonferroni should be applied. Say I have a research question for which I have three separate dependent variables, and run three separate models ...
151 views

### How is SNK.test ( or others ) doing groups in R?

I used the SNK.test (Student Newman Keuls Test) to compare different factor: ...
17 views

### Formulating hypothesys on continuous and categorial variables independence

Consider following process. We have $k$ people. They commit actions and we write down time of these actions in common list. We want to test the hypothesis that time of events is independent with who ...
38 views

### Formal definition of multiple hypothesis testing/multiple comparisons problem

I've been doing research on the topic of multiple comparisons, and most of the definitions I came across with, had the words multiple simultaneous statistical tests in them. Can anybody define ...
15 views

### Can you generate a 3-group ANOVA with significant F-ratio and all pairwise comparisons non-significant AND other restrictions?

This question is based on other questions and on my work writing statistics questions for online homework delivery systems.  (Happy to provide more info on this if requested.) I recognize ...
27 views

### experiment with 3 arms: analyzing results when one of 3 pairwise tests shows significance

I have an experiment with 3 groups: group 1 got no email group 2 got an email with 10 dollars incentive group 3 got an email with 20 dollars incentive to convert to a specific product. Conversion ...
111 views

### Multiple comparisons correction with independent groups

A question: I have a study with 3 experiments, each involving a between-subjects comparison of a treatment vs. control group. The groups in each experiment are independent samples and no groups are ...
82 views

### Does this situation necessitate multiple comparison corrections?

I am looking to run multiple t-tests on a dataset but I am unsure if the types of tests I am running would necessitate a multiple comparison correction or not. Below are essentially the tests I will ...
503 views

### Negative binomial pairwise comparison in R

I have fitted a negative binomial regression model to my data, and the summary of this compares latency of 3 resources to that of burrows: ...
31 views

### Do I need to “correct”?

I am testing whether a certain learning intervention affects student achievement. I have two groups of students (control and experimental) with 16 students each. I have their pre-test and post-test ...
44 views

### How to ı calculate sample size?

I am planning a study for comparing tear levels (which I will measure with the Schirmer test) between two groups. Group 1: persons with normal blood vitamin D levels and Group 2: persons with low ...
1k views

### Post Hoc Pairwise Comparison of Interaction in Mixed Effects (lmer) Model

I have searched other posts and textbooks and found numerous variations of the pairwise comparisions for my mixed effects model using code from the multicomp ...
295 views

### Regression when output is in a specific interval

Say my output is continuous on the interval $[-1, 1]$. Is there a standard technique to deal with this? Do I just use logistic function, same as in logistic regression, but for actual regression ...
622 views

### NA's appearing in my logistic regression summary output. How should I procede?

I have been following multiple posts on this same topic and understand why I have NA's in my summary table. I am wondering what is an appropriate course of action in order to conduct a post hoc ...
235 views

### Error when attempting post hoc analyses on a binomial glm with interactions

link to data I am having a problem conducting post hoc analyses on a binomial glm using the data linked above. Experiment: I am looking at the detectability (presence/absence) of an insect marker ...
74 views

### Is this an example of a multiple comparisons problem?

I am conducting a comprehensive research determining the effect of 3 independent variables on the metabolic rate of an organism. However, the set-up of my data analysis leads me to believe that I ...
75 views

### Is it appropriate to remove meaningless pairwise comparisons from a post-hoc analysis before correcting for multiple comparisons?

I have four individual groups with two controls and two interventions. The second control group is specifically related to the first the intervention group, making the comparison to the second ...
13 views

### Is there an analog of Benjamini-Hochberg for Bayesian hypothesis testing? [duplicate]

In Bayesian hypothesis testing, where one actually commits to estimates of the probabilities of each hypothesis, an analog of the Bonferroni correction is easy to justify. Namely, in the multiple ...
27 views

### Comparison of multiple means with different sample sizes

Given four classes, each with a different number of students: Class A: 41 students; Class B: 39 students; Class C: 38 students; Class D: 30 students; Each class took an exam and garnered the ...
187 views

### multiple correction comparisons (Interactions)

A reviewer asked me to do multiple comparisons corrections for two-way and three-way interactions that showed significance in my manuscript. I did my data analysis with linear mixed model in R. I ...
112 views

### p-value Inequality?

I was trying to go over this paper by Dudoit, Shaffer and Boldric when I came across this statement on page 78 (under section 2.4: Control of family-wise error rate): $P(P_j\leq x \vert H_j)\leq x,$ ...
128 views

### Sample size for multiple hypothesis test: Wilcoxon test and Benjamini–Hochberg procedure

I have an experiment with a control and multiple treatments and want to find which one performs the best. I would use the Wilcoxon pairwise test to test all the hypotheses and correct the p-values ...
82 views

### Why don't we control for multiple comparisons in factor analysis?

How can we trust the significance tests for any of our factor loadings, or even the correlations matrix, when so many tests are run in factor analysis? Especially for IRT models where we may be ...
371 views

### Is only Bonferrone but not Tukey HSD a way of correction for pairwise comparison?

What is the relationship between Bonferroni and Tukey HSD, are they just two methods of conduting corrections for post-hoc tests? If this is the case, could I say "post-hoc test was conducted using ...
46 views

### Does ordering the comparisons by p-value make sense?

In bioinformatics I usually do enrichment test (read hypergeometric test) with many comparisons over the same list. After correcting for multiple comparisons I usually order the results by p-value (...
93 views

### Permutation test, probability and FDR

I encountered a problem to perform FDR control of my data, and am not sure how can I do. I have hundreds to thousands of observations, each observation has two scores. So I have \$\mathbf{s_1}=\{s_{11}...
305 views

### Easy post-hoc tests when meta-analyzing with the metafor package in r

I am conducting meta-analysis using metafor in R. I would like to compare among 7 levels in a factor (i.e. different types of treatments). ...
277 views

### Which confidence interval adjustment should I do when using FDR p valures adjustment?

I need to do multiple comparison, and I want adjust the p-values by false discovery rate (fdr). However, it is impossible also adjust the confidence intervals by <...
337 views

### How to achieve a two-sided combined p-value using Fisher's method?

Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
97 views

### Is it appropriate to use Kruskal-Wallis test for one comparison and Wilcoxon Signed Rank Test for another?

I have a data set that I would like to perform several tests on to see if there are significant changes over time. I have replicated measures of 3 experimental ponds and 1 control pond that I have ...
390 views

### Multiple comparison correction (Bonferroni vs. Holm Boonferroni)

I have four questions: If i have to run a test on 4 colors; red, blue, green, yellow. I ran the same test 4 times (one time on each color). (H1~H4) If i already have a predefined or existing reason ...
902 views

### Correction of p-values for multiple regression models with multiple comparisons

I was asked to do a correction of p-values for my analyses, but I'm not sure what p-values I am to correct, and presuming I'm using Bonferroni, what is the number that I am dividing the .05 level with....
12 views

### Time series of car acceleration?

Study: Time series with 10 different timepoints, 10 different speeds for 4 different cars. I want to test if there are statistical differences between the accelerations of the cars. Can anybody tell ...
120 views

### Ranking K coins [closed]

How to rank coins without using p value?
708 views

### Post-Hoc pairwise comparison for slope (interaction term) in R?

I have been using scripts from package "Agricolae" but realised today that it compares only between the intercepts. I expect that since i am testing for the interaction term using ANCOVA, the post-...
22 views

### Which test should I use for the effect of a binary treatment on many continuous outcomes measured at 3 time points, controlling for baseline values?

I have a binary treatment, and 6 continuous outcomes (at least, they can take from 21 to 49 distinct values), all measured at baseline and at 3 time points after the treatment. I was thinking about ...
166 views

### How can I program correction for multiple comparisons using lsmeans in R while not correcting for more than necessary

I posted this question on stack overflow here enter link description here but did not get useful feedback. So, I'll post it again here, in the hopes that this is a better venue for it. I'm wondering ...
62 views

### Should missing observations be included in the number of observations if correcting for multiple testing

I am confused whether one should exclude missing observations when adjusting p values for multiple testing. There seems to be no consensus among R function on whether to do this or not. ...
375 views

### Should I use q-values or p-values when presenting results?

I am presenting an analysis in which I performed a number of tests for N variables. The variables which are significant at q < 0.05 are presented in a table (q is the Benjamini-Hochberg corrected p-...
28 views

### Is there a specific test for correlation between 2 measurable variables between nominal data?

I am having a hard time finding a comparable test for my data set. Here is what I have: weight of individuals and Concentration of a trace metal in each individual for 8 field sites. I want to see if ...
104 views

### Should I use post-hoc tukey HSD or some other test for pairwise comparisons of a factor on a zero-inflated negative binomial mixed model (ZINB)?

I ran a zero-inflated negative binomial mixed model (ZINB) and now have a statistically significant factor (recording type, either NOCA, BCCH, or Human). To assess the differences between these three ...
226 views

### Researcher 1 runs 1000 regressions, researcher 2 runs only 1, both get same results — should they make different inferences?

Imagine a researcher is exploring a dataset and runs 1000 different regressions and he finds one interesting relationship among them. Now imagine another researcher with the same data runs just 1 ...
730 views

### Post-Hoc Scheffe Test (Agricolae). Where are the coefficients?

I know this is fundamental. Please forgive me. I am new to R. I ran the Scheffe post-hoc test after ANCOVA using "scheffe.test" from package "Agricolae". I got the following output: ...
50 views

### Multiple Test Correction: robust against many insignificant p-values

I work in bioinformatics so I've seen my fair share of null multiple tests (figure 1) As well as a clear signal in p-value distributions (figure2) But I've also occasionally seen a type of p-value ...
49 views

### T-test of two groups of a 15 groups experiment

I'm a biologist, so for most of you my question is gonna be the dumbest one. I made an experiment with triplicates and performed anova and tukey's test, but the later was not significant. What I would ...
26 views

### Nonparametric test for 1 categorical IV and multiple binary DV:s?

I have a dataset where test subjects have answered three different questions (my IV). The answers to each question are scored on 10 different variables (presence or absence of 10 concepts in the ...
52 views

### Which is the best? Correct interpretation of paiwise comparison results

Consider the situation in which one should investigate, which treatment or treatments (of A, B, C, D, E, F, G, H) is/are the most effective (highest decrease) and which is/are the worst (lowest ...