Skip to main content

# Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1,867 questions
Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

### Use of multiple comparison correction and composite DV

I am investigating the effect of a single X on a composite DV, Y. I want to also investigate the effect of X on the 36 sub-components of Y. Thus, I would have to run 37 multiple regression models, ...
0 votes
1 answer
13 views

### Filter one-sided tests before FDR

In this blog post, the author shows one example of "weird" p-value histogram ("Scenario C"): One provided explanation is that a one-sided test was run, and the tests where the ...
• 155
1 vote
0 answers
22 views

### Recipes for combining multiple null-hypothesis tests

I have two questions about hypothesis testing: Is there one or multiple generic recipes for combining tests together? Some procedure that would produce a new better test $T_\star$ from two tests $T_1$...
• 2,516
2 votes
1 answer
32 views

### ANOVA finding a main effect prevents inflation of Type I error rate

In the (free) book Understanding Statistics and Experimental Design, section 6.3, they say '...ANOVA offers a second trick. If we rejected the null hypothesis, it is appropriate to compare pairs of ...
• 123
0 votes
1 answer
24 views

### What criterion to use to compare multiple correlations of binary variables?

I have $N$ definitions of certain properties of countries (for example, if the country is "democratic", "totalitarian" etc.), and want to test how consistently different people ...
0 votes
0 answers
26 views

### Appropriate way to correct for multiple comparisons in this case

I am trying to compare the performance of an optimisation algorithm on different objective functions. The functions can vary in their form (4 different types) and dimension. I would like to first ...
6 votes
1 answer
419 views

### Related low p-values that do not meet statistically significant thresholds

If I were to have multiple related measures, each with "low" p-values that do not quite meet the threshold of significance, could the fact that multiple measures produce a general trend with ...
0 votes
0 answers
21 views

### Impact of variation in the total number of samples when performing multiple fisher exact tests

I have dataset of 2 time points. In each time point I have to compare the counts of different components at each position (number of positions =1000). I used fisher exact test to compare the if there ...
0 votes
0 answers
46 views

### Steps to conduct a linear mixed model and post-hoc comparison test

I would like to ask for your help with the best way to analyze the following experiment within R. Here is my design: there are 15 treatments, in four blocks. For each treatment, in each block, we have ...
2 votes
2 answers
45 views

### Bonferroni correction necessary? regression to predict ONE outcome

I am working on a project with a small sample size where I have multiple predictors at baseline and one IV. I am trying to see if any of the DVs are good predictors for the score on the IV (continuous)...
0 votes
0 answers
12 views

### Post-hoc tests for repeated measures ANOVA with aligned rank transformed(ART) data

I'm working on a 2x3 factorial design with two within-subject variables and I'd like to test for both main and interaction effects. Since the data is not normally distributed, I've performed an ...
4 votes
2 answers
67 views

### Making 3 comparisons with a ternary variable

I have a study in which I want to study the effect of a ternary variable $x$ in interaction with a binary variable $y$. The ternary variable is categorical, and for some reasons, I want to do the $3$ ...
1 vote
1 answer
50 views

### Multiple comparisons and explanation of two-factor experiment

I am doing investigation with temperature and accession on trait of plants with R. Temperature: 25/20, 30/25, 35/30 Accession: V9,BTx623, 2-16-6, 1-10-4, 2-16-6, 19-22-12 The interaction and ...
• 123
2 votes
2 answers
194 views

1 vote
0 answers
71 views

### Statistical significance of hierarchical (multi-level) associations

Association rule mining is a common technique in Data mining used to find itemsets which appear together very frequently in large transaction databases (e.g. in Table 3 we see that in a transactions ...
0 votes
0 answers
10 views

### Methods for discriminating between Markov kernels

I'm interested in problems of the following form, which I've deliberately specified a bit vaguely. I would like to know if problems of this kind have been studied, and if so what they're called and ...
• 425
6 votes
1 answer
182 views

### How to adjust p-values in a multiple regression when variables have been a selected by Lasso?

I have designed an analysis where I am testing a lot of variables together. So I first apply a Lasso regression to select the top variables, and then I run a standard (unregularized) multiple ...
1 vote
1 answer
133 views

### multiple testing adjustment for fisher exact test

I have 4 groups (G1, G2, G3, G4) consisting of samples of biological material, and we test them in order to identify are there any bacteria present. We also know that all of the samples contain virus ...
1 vote
1 answer
22 views

### Group Comparison with serial mediation

I'm trying to compare two groups for a serial mediation model. I wrote my code like in this link to make it clearer, here is my code: ...
• 21
2 votes
1 answer
94 views

### Am I using emmeans correctly to look at pairwise differences after running a type III ANOVA model?

I am looking at how the weight of female individuals is influenced by the two factors A (5 levels) and B (2 levels) using a type III ANOVA as follows: ...
2 votes
0 answers
39 views

### Sequential analysis - adjusting effect size

I am aware that sequential testing is risky, but it makes sense for me to do it anyway. So, I would like to read up, to do it properly. My idea (which is probably not new): In my reality, the true ...
• 339
0 votes
0 answers
27 views

### Type I error control for multiple comparisons

In statistics, there is approach, aka, fixed-sequence hirarchical approach to control the Type I error when it comes to multiple tests for different variables. For example, there are 3 tests to be ...
• 65
0 votes
0 answers
21 views

### How to do ANOVA/post-hoc analysis when 'control' is used to normalise the data?

I am analysing the transportation rate for a number of different mutants in a particular transportation system. I want to know how I can perform statistical tests on this data in a valid way, given ...
1 vote
0 answers
10 views

### Pairwise trend comparisons for a joint model

I have built a joint model for longitudinal and survival data using the jointModel function in JM. I would like to perform pairwise comparisons of three trends of the longitudinal outcome of the joint ...
• 1,109
0 votes
0 answers
28 views

### What really happens if I run Tukey without running ANOVA first?

I see that the theory says that if I want to test for difference in mean between 3+ groups I should use ANOVA first, and if ANOVA rejects the null hypothesis then I might want to look into Tukey as a ...
• 566