Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1
vote
1answer
25 views

Implications of descriptive statistics on parameter estimates

I am working on my dissertation and a question dawned on me after I ran my analyses. I studied whether the interaction between marital conflict and gratitude on mental health differed between first-...
0
votes
0answers
37 views

How to use Bonferroni correction - correct by no of parameters? No of models per dependent variable?

I have a simple question about when corrections such as Bonferroni should be applied. Say I have a research question for which I have three separate dependent variables, and run three separate models ...
0
votes
1answer
151 views

How is SNK.test ( or others ) doing groups in R?

I used the SNK.test (Student Newman Keuls Test) to compare different factor: ...
1
vote
0answers
17 views

Formulating hypothesys on continuous and categorial variables independence

Consider following process. We have $k$ people. They commit actions and we write down time of these actions in common list. We want to test the hypothesis that time of events is independent with who ...
1
vote
1answer
38 views

Formal definition of multiple hypothesis testing/multiple comparisons problem

I've been doing research on the topic of multiple comparisons, and most of the definitions I came across with, had the words multiple simultaneous statistical tests in them. Can anybody define ...
0
votes
0answers
15 views

Can you generate a 3-group ANOVA with significant F-ratio and all pairwise comparisons non-significant AND other restrictions?

This question is based on other questions and on my work writing statistics questions for online homework delivery systems.  (Happy to provide more info on this if requested.) I recognize ...
0
votes
2answers
27 views

experiment with 3 arms: analyzing results when one of 3 pairwise tests shows significance

I have an experiment with 3 groups: group 1 got no email group 2 got an email with 10 dollars incentive group 3 got an email with 20 dollars incentive to convert to a specific product. Conversion ...
3
votes
2answers
111 views

Multiple comparisons correction with independent groups

A question: I have a study with 3 experiments, each involving a between-subjects comparison of a treatment vs. control group. The groups in each experiment are independent samples and no groups are ...
0
votes
3answers
82 views

Does this situation necessitate multiple comparison corrections?

I am looking to run multiple t-tests on a dataset but I am unsure if the types of tests I am running would necessitate a multiple comparison correction or not. Below are essentially the tests I will ...
3
votes
1answer
503 views

Negative binomial pairwise comparison in R

I have fitted a negative binomial regression model to my data, and the summary of this compares latency of 3 resources to that of burrows: ...
0
votes
0answers
31 views

Do I need to “correct”?

I am testing whether a certain learning intervention affects student achievement. I have two groups of students (control and experimental) with 16 students each. I have their pre-test and post-test ...
0
votes
1answer
44 views

How to ı calculate sample size?

I am planning a study for comparing tear levels (which I will measure with the Schirmer test) between two groups. Group 1: persons with normal blood vitamin D levels and Group 2: persons with low ...
1
vote
1answer
1k views

Post Hoc Pairwise Comparison of Interaction in Mixed Effects (lmer) Model

I have searched other posts and textbooks and found numerous variations of the pairwise comparisions for my mixed effects model using code from the multicomp ...
5
votes
2answers
295 views

Regression when output is in a specific interval

Say my output is continuous on the interval $[-1, 1]$. Is there a standard technique to deal with this? Do I just use logistic function, same as in logistic regression, but for actual regression ...
1
vote
0answers
622 views

NA's appearing in my logistic regression summary output. How should I procede?

I have been following multiple posts on this same topic and understand why I have NA's in my summary table. I am wondering what is an appropriate course of action in order to conduct a post hoc ...
0
votes
0answers
235 views

Error when attempting post hoc analyses on a binomial glm with interactions

link to data I am having a problem conducting post hoc analyses on a binomial glm using the data linked above. Experiment: I am looking at the detectability (presence/absence) of an insect marker ...
0
votes
3answers
74 views

Is this an example of a multiple comparisons problem?

I am conducting a comprehensive research determining the effect of 3 independent variables on the metabolic rate of an organism. However, the set-up of my data analysis leads me to believe that I ...
3
votes
1answer
75 views

Is it appropriate to remove meaningless pairwise comparisons from a post-hoc analysis before correcting for multiple comparisons?

I have four individual groups with two controls and two interventions. The second control group is specifically related to the first the intervention group, making the comparison to the second ...
2
votes
0answers
13 views

Is there an analog of Benjamini-Hochberg for Bayesian hypothesis testing? [duplicate]

In Bayesian hypothesis testing, where one actually commits to estimates of the probabilities of each hypothesis, an analog of the Bonferroni correction is easy to justify. Namely, in the multiple ...
0
votes
1answer
27 views

Comparison of multiple means with different sample sizes

Given four classes, each with a different number of students: Class A: 41 students; Class B: 39 students; Class C: 38 students; Class D: 30 students; Each class took an exam and garnered the ...
2
votes
1answer
187 views

multiple correction comparisons (Interactions)

A reviewer asked me to do multiple comparisons corrections for two-way and three-way interactions that showed significance in my manuscript. I did my data analysis with linear mixed model in R. I ...
2
votes
1answer
112 views

p-value Inequality?

I was trying to go over this paper by Dudoit, Shaffer and Boldric when I came across this statement on page 78 (under section 2.4: Control of family-wise error rate): $P(P_j\leq x \vert H_j)\leq x,$ ...
1
vote
0answers
128 views

Sample size for multiple hypothesis test: Wilcoxon test and Benjamini–Hochberg procedure

I have an experiment with a control and multiple treatments and want to find which one performs the best. I would use the Wilcoxon pairwise test to test all the hypotheses and correct the p-values ...
3
votes
1answer
82 views

Why don't we control for multiple comparisons in factor analysis?

How can we trust the significance tests for any of our factor loadings, or even the correlations matrix, when so many tests are run in factor analysis? Especially for IRT models where we may be ...
1
vote
1answer
371 views

Is only Bonferrone but not Tukey HSD a way of correction for pairwise comparison?

What is the relationship between Bonferroni and Tukey HSD, are they just two methods of conduting corrections for post-hoc tests? If this is the case, could I say "post-hoc test was conducted using ...
2
votes
0answers
46 views

Does ordering the comparisons by p-value make sense?

In bioinformatics I usually do enrichment test (read hypergeometric test) with many comparisons over the same list. After correcting for multiple comparisons I usually order the results by p-value (...
1
vote
0answers
93 views

Permutation test, probability and FDR

I encountered a problem to perform FDR control of my data, and am not sure how can I do. I have hundreds to thousands of observations, each observation has two scores. So I have $\mathbf{s_1}=\{s_{11}...
3
votes
1answer
305 views

Easy post-hoc tests when meta-analyzing with the `metafor` package in r

I am conducting meta-analysis using metafor in R. I would like to compare among 7 levels in a factor (i.e. different types of treatments). ...
2
votes
1answer
277 views

Which confidence interval adjustment should I do when using FDR p valures adjustment?

I need to do multiple comparison, and I want adjust the p-values by false discovery rate (fdr). However, it is impossible also adjust the confidence intervals by <...
1
vote
1answer
337 views

How to achieve a two-sided combined p-value using Fisher's method?

Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
1
vote
0answers
97 views

Is it appropriate to use Kruskal-Wallis test for one comparison and Wilcoxon Signed Rank Test for another?

I have a data set that I would like to perform several tests on to see if there are significant changes over time. I have replicated measures of 3 experimental ponds and 1 control pond that I have ...
2
votes
1answer
390 views

Multiple comparison correction (Bonferroni vs. Holm Boonferroni)

I have four questions: If i have to run a test on 4 colors; red, blue, green, yellow. I ran the same test 4 times (one time on each color). (H1~H4) If i already have a predefined or existing reason ...
1
vote
1answer
902 views

Correction of p-values for multiple regression models with multiple comparisons

I was asked to do a correction of p-values for my analyses, but I'm not sure what p-values I am to correct, and presuming I'm using Bonferroni, what is the number that I am dividing the .05 level with....
1
vote
0answers
12 views

Time series of car acceleration?

Study: Time series with 10 different timepoints, 10 different speeds for 4 different cars. I want to test if there are statistical differences between the accelerations of the cars. Can anybody tell ...
0
votes
1answer
120 views

Ranking K coins [closed]

How to rank coins without using p value?
1
vote
1answer
708 views

Post-Hoc pairwise comparison for slope (interaction term) in R?

I have been using scripts from package "Agricolae" but realised today that it compares only between the intercepts. I expect that since i am testing for the interaction term using ANCOVA, the post-...
0
votes
0answers
22 views

Which test should I use for the effect of a binary treatment on many continuous outcomes measured at 3 time points, controlling for baseline values?

I have a binary treatment, and 6 continuous outcomes (at least, they can take from 21 to 49 distinct values), all measured at baseline and at 3 time points after the treatment. I was thinking about ...
3
votes
2answers
166 views

How can I program correction for multiple comparisons using lsmeans in R while not correcting for more than necessary

I posted this question on stack overflow here enter link description here but did not get useful feedback. So, I'll post it again here, in the hopes that this is a better venue for it. I'm wondering ...
4
votes
1answer
62 views

Should missing observations be included in the number of observations if correcting for multiple testing

I am confused whether one should exclude missing observations when adjusting p values for multiple testing. There seems to be no consensus among R function on whether to do this or not. ...
3
votes
2answers
375 views

Should I use q-values or p-values when presenting results?

I am presenting an analysis in which I performed a number of tests for N variables. The variables which are significant at q < 0.05 are presented in a table (q is the Benjamini-Hochberg corrected p-...
1
vote
0answers
28 views

Is there a specific test for correlation between 2 measurable variables between nominal data?

I am having a hard time finding a comparable test for my data set. Here is what I have: weight of individuals and Concentration of a trace metal in each individual for 8 field sites. I want to see if ...
2
votes
0answers
104 views

Should I use post-hoc tukey HSD or some other test for pairwise comparisons of a factor on a zero-inflated negative binomial mixed model (ZINB)?

I ran a zero-inflated negative binomial mixed model (ZINB) and now have a statistically significant factor (recording type, either NOCA, BCCH, or Human). To assess the differences between these three ...
12
votes
3answers
226 views

Researcher 1 runs 1000 regressions, researcher 2 runs only 1, both get same results — should they make different inferences?

Imagine a researcher is exploring a dataset and runs 1000 different regressions and he finds one interesting relationship among them. Now imagine another researcher with the same data runs just 1 ...
0
votes
1answer
730 views

Post-Hoc Scheffe Test (Agricolae). Where are the coefficients?

I know this is fundamental. Please forgive me. I am new to R. I ran the Scheffe post-hoc test after ANCOVA using "scheffe.test" from package "Agricolae". I got the following output: ...
1
vote
1answer
50 views

Multiple Test Correction: robust against many insignificant p-values

I work in bioinformatics so I've seen my fair share of null multiple tests (figure 1) As well as a clear signal in p-value distributions (figure2) But I've also occasionally seen a type of p-value ...
0
votes
1answer
49 views

T-test of two groups of a 15 groups experiment

I'm a biologist, so for most of you my question is gonna be the dumbest one. I made an experiment with triplicates and performed anova and tukey's test, but the later was not significant. What I would ...
1
vote
0answers
26 views

Nonparametric test for 1 categorical IV and multiple binary DV:s?

I have a dataset where test subjects have answered three different questions (my IV). The answers to each question are scored on 10 different variables (presence or absence of 10 concepts in the ...
2
votes
0answers
52 views

Which is the best? Correct interpretation of paiwise comparison results

Consider the situation in which one should investigate, which treatment or treatments (of A, B, C, D, E, F, G, H) is/are the most effective (highest decrease) and which is/are the worst (lowest ...
2
votes
0answers
225 views

Do Tukey and Games-Howell tests need additional p value corrections?

I was reading documentation of function posthocTGH (the "Note" section, where authors expressed doubt about additional corrections to p-values) and linked ...
0
votes
0answers
19 views

Analyzing multiple yes/no features of a sample

What will be the best statistical way to analyze presence/absence of multiple discrete variables for a given sample. Following is example data which shows 4 variables, each of which can be only Yes ...