Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
19 views

Comparing results of a promotion camapign: Control vs Treatment Sub-Group

I'm looking for some clarification on the correct way to test the difference between groups. At my company we ran a promotional campaign for credit card customers with a chance to win a prize. We took ...
0
votes
0answers
181 views

Error when attempting post hoc analyses on a binomial glm with interactions

link to data I am having a problem conducting post hoc analyses on a binomial glm using the data linked above. Experiment: I am looking at the detectability (presence/absence) of an insect marker ...
0
votes
0answers
55 views

Is there a normal assumption for Benjamini & Yekutieli (2001) false discovery rate procedure?

This question works its way through a lot of the differences between Benjamini & Yekutieli (BY) (2001) and the Benjamini & Hochberg (BH) procedure. It's my understanding that "BY" works well ...
0
votes
0answers
13 views

Which statistical test use to compare output of multiple voters?

Assume following situation. We have 5 voters, every of them picks up exactly 10 preferred candidates from pool of 60 candidates. We can assume that the order of candidates does not matter. So far ...
0
votes
0answers
16 views

Subsetting a dataset for testing prioritization

I often run into the scenario where I am testing whether a large number of features individually differ in abundance between sample groupings. For instance, many t-tests run for whether variables ...
0
votes
3answers
69 views

Is this an example of a multiple comparisons problem?

I am conducting a comprehensive research determining the effect of 3 independent variables on the metabolic rate of an organism. However, the set-up of my data analysis leads me to believe that I ...
0
votes
0answers
17 views

What tests to run?

I have rats that received a 30 minute infusion of control (saline) followed by 30 minutes of a low, mid, and high dose (total 120 minutes). A measurement (heart rate) was taken once during each dose ...
0
votes
0answers
87 views

Same p-values after Benjamini-Hochberg adjustment on simultaneous Tukey contrast tests

I'm running 7 different simultaneous logistic regressions on the same dataset. There is only one independent variable and 7 dependent variable. IV has 5 levels (it is a diagnostic score in MRI; as....
3
votes
1answer
65 views

Is it appropriate to remove meaningless pairwise comparisons from a post-hoc analysis before correcting for multiple comparisons?

I have four individual groups with two controls and two interventions. The second control group is specifically related to the first the intervention group, making the comparison to the second ...
2
votes
0answers
12 views

Is there an analog of Benjamini-Hochberg for Bayesian hypothesis testing? [duplicate]

In Bayesian hypothesis testing, where one actually commits to estimates of the probabilities of each hypothesis, an analog of the Bonferroni correction is easy to justify. Namely, in the multiple ...
0
votes
0answers
14 views

Requirements for multiple comparison correction

It is usual practice in certain fields to report socio-demographic data for different groups (say two) and then do a hypothesis test (like t test) to check if there is any significant difference ...
0
votes
0answers
16 views

What is the correct statistical test to use in order to find difference using multiple comparisons?

I am struggling to find the correct statistical test in order to compare two cell lines. The experiment is preformed in a 96-well plate and includes the addition of an antibiotics in order to produce ...
0
votes
1answer
24 views

Comparison of multiple means with different sample sizes

Given four classes, each with a different number of students: Class A: 41 students; Class B: 39 students; Class C: 38 students; Class D: 30 students; Each class took an exam and garnered the ...
2
votes
1answer
132 views

multiple correction comparisons (Interactions)

A reviewer asked me to do multiple comparisons corrections for two-way and three-way interactions that showed significance in my manuscript. I did my data analysis with linear mixed model in R. I ...
2
votes
1answer
91 views

p-value Inequality?

I was trying to go over this paper by Dudoit, Shaffer and Boldric when I came across this statement on page 78 (under section 2.4: Control of family-wise error rate): $P(P_j\leq x \vert H_j)\leq x,$ ...
0
votes
0answers
35 views

What is the best statistical test for comparing a binomial outcome variable across different conditions?

My design is the following: I have two main tasks and they are within-subjects. Task one is characterised by 1 dependent variable (DV1= ratings) and 1 independent variable (IV1) on 2 levels (the two ...
1
vote
0answers
101 views

Sample size for multiple hypothesis test: Wilcoxon test and Benjamini–Hochberg procedure

I have an experiment with a control and multiple treatments and want to find which one performs the best. I would use the Wilcoxon pairwise test to test all the hypotheses and correct the p-values ...
3
votes
1answer
68 views

Why don't we control for multiple comparisons in factor analysis?

How can we trust the significance tests for any of our factor loadings, or even the correlations matrix, when so many tests are run in factor analysis? Especially for IRT models where we may be ...
0
votes
0answers
114 views

Post-hoc comparison after GzLM Poisson Regression in MATLAB

I am currently performing a regression analysis between a count dependent variable and a categorical independent variable, controlling for the effects of a covariate. The most simple way is to perform ...
0
votes
0answers
23 views

3D data - significance testing and model fitting

My data are 3 dimensional. X and Y are independent variables, and Z is the (stochastic) dependent variable. I will take multiple (~15) measurements of Z at each X-Y point, so this would be fuzzy in ...
1
vote
1answer
321 views

Is only Bonferrone but not Tukey HSD a way of correction for pairwise comparison?

What is the relationship between Bonferroni and Tukey HSD, are they just two methods of conduting corrections for post-hoc tests? If this is the case, could I say "post-hoc test was conducted using ...
2
votes
0answers
45 views

Does ordering the comparisons by p-value make sense?

In bioinformatics I usually do enrichment test (read hypergeometric test) with many comparisons over the same list. After correcting for multiple comparisons I usually order the results by p-value (...
1
vote
0answers
80 views

Permutation test, probability and FDR

I encountered a problem to perform FDR control of my data, and am not sure how can I do. I have hundreds to thousands of observations, each observation has two scores. So I have $\mathbf{s_1}=\{s_{11}...
3
votes
1answer
264 views

Easy post-hoc tests when meta-analyzing with the `metafor` package in r

I am conducting meta-analysis using metafor in R. I would like to compare among 7 levels in a factor (i.e. different types of treatments). ...
2
votes
1answer
243 views

Which confidence interval adjustment should I do when using FDR p valures adjustment?

I need to do multiple comparison, and I want adjust the p-values by false discovery rate (fdr). However, it is impossible also adjust the confidence intervals by <...
0
votes
0answers
20 views

How to determine the sample size for multimodal distributions or mixture of normals

A simulation provides non-deterministic results. These results follow different distributions functions depending on the input parameters. The goals is to categorize the resulting distributions based ...
1
vote
1answer
288 views

How to achieve a two-sided combined p-value using Fisher's method?

Let's say I want to answer the question if smokers vs nonsmokers have different levels of Gene A. This seems like an obvious two-sided test. However, if I have multiple studies and I want to combine ...
0
votes
0answers
17 views

is a sample size of 10 in each group (5 groups) statistically follows non-parametric tests regardless the data are normally disrtibuted or not?

I'm working on a project in which I have 10 rats in each group (5 groups) The results I got were subjected to SPSS program for the test of normality and the data were normally distributed. I was ...
1
vote
0answers
87 views

Is it appropriate to use Kruskal-Wallis test for one comparison and Wilcoxon Signed Rank Test for another?

I have a data set that I would like to perform several tests on to see if there are significant changes over time. I have replicated measures of 3 experimental ponds and 1 control pond that I have ...
2
votes
1answer
288 views

Multiple comparison correction (Bonferroni vs. Holm Boonferroni)

I have four questions: If i have to run a test on 4 colors; red, blue, green, yellow. I ran the same test 4 times (one time on each color). (H1~H4) If i already have a predefined or existing reason ...
0
votes
0answers
28 views

Testing if different coins are equal

I have 5 different coins. I would like to test if they're equal. I know how to test two coins at the same time, using a z-test. This case is different because I'm referring to multiple coins. In ...
1
vote
1answer
732 views

Correction of p-values for multiple regression models with multiple comparisons

I was asked to do a correction of p-values for my analyses, but I'm not sure what p-values I am to correct, and presuming I'm using Bonferroni, what is the number that I am dividing the .05 level with....
1
vote
0answers
11 views

Time series of car acceleration?

Study: Time series with 10 different timepoints, 10 different speeds for 4 different cars. I want to test if there are statistical differences between the accelerations of the cars. Can anybody tell ...
0
votes
0answers
28 views

What is the proper way to analyze data in which the control condition has no variance (because it was set to 1 in each experiment)?

I would appreciate the community's help: I'm analyzing protein expression by western blot and setting my control condition to equal to 1 (I need to do this based on my method of densitometry). When I ...
0
votes
1answer
120 views

Ranking K coins [closed]

How to rank coins without using p value?
0
votes
1answer
617 views

Post-Hoc pairwise comparison for slope (interaction term) in R?

I have been using scripts from package "Agricolae" but realised today that it compares only between the intercepts. I expect that since i am testing for the interaction term using ANCOVA, the post-...
0
votes
0answers
22 views

Which test should I use for the effect of a binary treatment on many continuous outcomes measured at 3 time points, controlling for baseline values?

I have a binary treatment, and 6 continuous outcomes (at least, they can take from 21 to 49 distinct values), all measured at baseline and at 3 time points after the treatment. I was thinking about ...
0
votes
0answers
16 views

Using PCA before multiple series test to solve dependence structure issue

In controlling method of FWE(family-wise error), researchers came up with new ways to account for true dependence structure of the P values. For an example, on single step correction, we evolved from ...
0
votes
0answers
247 views

Do multiple bootstrap generated p-values need to be corrected for multiple tests?

  tl;dr do I need to correct p-values from multiple bootstraps? I'm finding seemingly conflicting results form previous answers here and here. I have a collection of 100 ordered samples, that ...
3
votes
2answers
123 views

How can I program correction for multiple comparisons using lsmeans in R while not correcting for more than necessary

I posted this question on stack overflow here enter link description here but did not get useful feedback. So, I'll post it again here, in the hopes that this is a better venue for it. I'm wondering ...
4
votes
1answer
56 views

Should missing observations be included in the number of observations if correcting for multiple testing

I am confused whether one should exclude missing observations when adjusting p values for multiple testing. There seems to be no consensus among R function on whether to do this or not. ...
3
votes
2answers
311 views

Should I use q-values or p-values when presenting results?

I am presenting an analysis in which I performed a number of tests for N variables. The variables which are significant at q < 0.05 are presented in a table (q is the Benjamini-Hochberg corrected p-...
1
vote
0answers
28 views

Is there a specific test for correlation between 2 measurable variables between nominal data?

I am having a hard time finding a comparable test for my data set. Here is what I have: weight of individuals and Concentration of a trace metal in each individual for 8 field sites. I want to see if ...
2
votes
0answers
95 views

Should I use post-hoc tukey HSD or some other test for pairwise comparisons of a factor on a zero-inflated negative binomial mixed model (ZINB)?

I ran a zero-inflated negative binomial mixed model (ZINB) and now have a statistically significant factor (recording type, either NOCA, BCCH, or Human). To assess the differences between these three ...
12
votes
3answers
225 views

Researcher 1 runs 1000 regressions, researcher 2 runs only 1, both get same results — should they make different inferences?

Imagine a researcher is exploring a dataset and runs 1000 different regressions and he finds one interesting relationship among them. Now imagine another researcher with the same data runs just 1 ...
0
votes
1answer
590 views

Post-Hoc Scheffe Test (Agricolae). Where are the coefficients?

I know this is fundamental. Please forgive me. I am new to R. I ran the Scheffe post-hoc test after ANCOVA using "scheffe.test" from package "Agricolae". I got the following output: ...
1
vote
1answer
47 views

Multiple Test Correction: robust against many insignificant p-values

I work in bioinformatics so I've seen my fair share of null multiple tests (figure 1) As well as a clear signal in p-value distributions (figure2) But I've also occasionally seen a type of p-value ...
0
votes
1answer
47 views

T-test of two groups of a 15 groups experiment

I'm a biologist, so for most of you my question is gonna be the dumbest one. I made an experiment with triplicates and performed anova and tukey's test, but the later was not significant. What I would ...
1
vote
0answers
26 views

Nonparametric test for 1 categorical IV and multiple binary DV:s?

I have a dataset where test subjects have answered three different questions (my IV). The answers to each question are scored on 10 different variables (presence or absence of 10 concepts in the ...
2
votes
0answers
46 views

Which is the best? Correct interpretation of paiwise comparison results

Consider the situation in which one should investigate, which treatment or treatments (of A, B, C, D, E, F, G, H) is/are the most effective (highest decrease) and which is/are the worst (lowest ...