Skip to main content

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

Use of multiple comparison correction and composite DV

I am investigating the effect of a single X on a composite DV, Y. I want to also investigate the effect of X on the 36 sub-components of Y. Thus, I would have to run 37 multiple regression models, ...
taylor's user avatar
  • 1
0 votes
1 answer
13 views

Filter one-sided tests before FDR

In this blog post, the author shows one example of "weird" p-value histogram ("Scenario C"): One provided explanation is that a one-sided test was run, and the tests where the ...
Alexlok's user avatar
  • 155
1 vote
0 answers
22 views

Recipes for combining multiple null-hypothesis tests

I have two questions about hypothesis testing: Is there one or multiple generic recipes for combining tests together? Some procedure that would produce a new better test $T_\star$ from two tests $T_1$...
Guillaume Dehaene's user avatar
2 votes
1 answer
32 views

ANOVA finding a main effect prevents inflation of Type I error rate

In the (free) book Understanding Statistics and Experimental Design, section 6.3, they say '...ANOVA offers a second trick. If we rejected the null hypothesis, it is appropriate to compare pairs of ...
Quinn Culver's user avatar
0 votes
1 answer
24 views

What criterion to use to compare multiple correlations of binary variables?

I have $N$ definitions of certain properties of countries (for example, if the country is "democratic", "totalitarian" etc.), and want to test how consistently different people ...
Daigaku no Baku's user avatar
0 votes
0 answers
26 views

Appropriate way to correct for multiple comparisons in this case

I am trying to compare the performance of an optimisation algorithm on different objective functions. The functions can vary in their form (4 different types) and dimension. I would like to first ...
ufghd34's user avatar
6 votes
1 answer
419 views

Related low p-values that do not meet statistically significant thresholds

If I were to have multiple related measures, each with "low" p-values that do not quite meet the threshold of significance, could the fact that multiple measures produce a general trend with ...
Jaxton Winder's user avatar
0 votes
0 answers
21 views

Impact of variation in the total number of samples when performing multiple fisher exact tests

I have dataset of 2 time points. In each time point I have to compare the counts of different components at each position (number of positions =1000). I used fisher exact test to compare the if there ...
Marwah Al-kaabi's user avatar
0 votes
0 answers
46 views

Steps to conduct a linear mixed model and post-hoc comparison test

I would like to ask for your help with the best way to analyze the following experiment within R. Here is my design: there are 15 treatments, in four blocks. For each treatment, in each block, we have ...
Caio Mattos Pereira's user avatar
2 votes
2 answers
45 views

Bonferroni correction necessary? regression to predict ONE outcome

I am working on a project with a small sample size where I have multiple predictors at baseline and one IV. I am trying to see if any of the DVs are good predictors for the score on the IV (continuous)...
user418492's user avatar
0 votes
0 answers
12 views

Post-hoc tests for repeated measures ANOVA with aligned rank transformed(ART) data

I'm working on a 2x3 factorial design with two within-subject variables and I'd like to test for both main and interaction effects. Since the data is not normally distributed, I've performed an ...
fr000g's user avatar
  • 1
4 votes
2 answers
67 views

Making 3 comparisons with a ternary variable

I have a study in which I want to study the effect of a ternary variable $x$ in interaction with a binary variable $y$. The ternary variable is categorical, and for some reasons, I want to do the $3$ ...
Alexandra Steinhilber's user avatar
1 vote
1 answer
50 views

Multiple comparisons and explanation of two-factor experiment

I am doing investigation with temperature and accession on trait of plants with R. Temperature: 25/20, 30/25, 35/30 Accession: V9,BTx623, 2-16-6, 1-10-4, 2-16-6, 19-22-12 The interaction and ...
Ken Hsu's user avatar
  • 123
2 votes
2 answers
194 views

How to compare two PMF generating functions?

I have two functions $f$ and $g$ that map an input vector $x_i$ to a probability mass function. For example: $$ x_i = [0,1,2] \\\\ f(x_i) \rightarrow [0.1, 0.3, 0.2, 0.2, 0.2] \\\\ g(x_i) \...
ilija139's user avatar
0 votes
0 answers
31 views

GLM Multiple Comparisons

I am performing several Generalized Linear Models in my analysis and I am wondering which method to use for adjusting p-values due to multiple comparisons. I have 4 outcomes (judgement of intensity ...
KayAnn's user avatar
  • 11
2 votes
1 answer
37 views

"paired" comparison of more than two algorithms

When comparing two algorithms $A$ and $B$ on test data $x_1,\ldots,x_n$ by means of a performance measure that yields $A(x_i)$ and $B(x_i)$, there are two standard methods: Check whether $\frac{1}{n}\...
cdalitz's user avatar
  • 5,392
0 votes
0 answers
14 views

When is Tukey post hoc test result identical to Games Howell post hoc test for Welch ANOVA?

In what situations is the Tukey post hoc test identical to the Games Howell post hoc test? I have an ANOVA analysis where HOV is violated and I have a statistically significant Welch ANOVA result. I ...
L.S.'s user avatar
  • 341
0 votes
0 answers
16 views

How to report few-shot accuracy for LLMs?

I am comparing three prompting techniques in LLMs to check which one is best. All prompting strategies include three examples for in-context learning (few-shot only, no fine-tuning). If I do greedy ...
Jader Martins's user avatar
6 votes
2 answers
309 views

Is it multiple comparisons?

Our study hypothesis is that metabolic syndrome (MetS) has impact on some outcome. There are at least five definitions of MetS in the literature. One of colleagues suggested using all five definitions ...
 Ivan.S's user avatar
1 vote
1 answer
34 views

Mixed Effects with repeated measures design and covariate included

I have a dataset that consists of the oxygen consumption (MO2) values for 3 different species of fish. My experiment was a repeated measures design where I measured MO2 of the same 14 fish per species ...
Allison's user avatar
  • 31
3 votes
1 answer
35 views

Significant results from ANOVA post-hoc Tukey's, insignificant with welch-ANOVA post-hoc Dunnett's T3

I performed one-way ANOVA tests with post-hoc Tukey's to look at multiple comparisons and got highly significant results, but noticed the $F$-value on the ANOVA was high. The SDs in the groups were ...
Harry's user avatar
  • 31
2 votes
1 answer
62 views

Is it possible to do statistic analysis from only knowing my groups's means and sample sizes?

I am writing about a problem my colleagues (in human health field) consider impossible to solve, but I want to believe mathematics have an answer. I am being told it is impossible to do statistical ...
Melissa Gaultier's user avatar
3 votes
1 answer
53 views

Identical SE values for different groups using emtrends

I'm using emtrends to extract slopes and do pairwise comparisons between the groups of my independent variable, controlled for the other variables in the model. ...
Mike's user avatar
  • 131
0 votes
1 answer
44 views

Which statistical test should I use for 3 independent variables and one dependent variable?

What are the tests that I can use for 3 independent variables (ordinal - Likert) and 1 dependent variable (college grades - general average)? I am trying to see if the IV really affects their GWA (DV) ...
Emma's user avatar
  • 3
0 votes
0 answers
29 views

Why does the Dunn's test in online calculator rejects the null hypothesis while my manual computations cannot reject it?

In my own manual calculations of the Dunn's Test in google sheet, using the formula from a YouTube video, I managed to calculate the same Test Statistic as the online calculator. However, using the ...
FifthRevelathor's user avatar
1 vote
1 answer
41 views

How do I do the p-adjustement in statistical tests?

context: a study about which type of bread and/or filling attracts ants the most I have a table with 3 columns : the number of ants attracted, type of bread, type of filling. The question is : do ...
Moez Daly's user avatar
2 votes
2 answers
54 views

type I error in multiple comparisons summary statistics

I understand type I error can increase if we run 'multiple comparisons'. But does that refer to comparisons within a multi-category variable (or a group variable)? or multiple analyses using the same ...
aqen's user avatar
  • 191
0 votes
0 answers
21 views

Multiple comparison tests when removing the intercept?

I have a set of 13 different dependent variables, all tested against 9 independent variables. Here's a simplified example of the tests I'm running using R syntax: ...
millie0725's user avatar
0 votes
1 answer
25 views

validity of multiple comparisons

Suppose I have 2 conditions, A and B, and I estimate 10 different metrics in each condition. If I want to test whether each metric is differentiated across the two conditions, do I have to control for ...
Paris's user avatar
  • 21
1 vote
1 answer
18 views

Combining Z scores of Correlated Test Results

I have the Z scores of a population of N for M different tests, where N is O(10000) and M is ~50. The results of these test are not independent, but we can produce a reasonable estimate of the ...
AspiringStatistician's user avatar
1 vote
1 answer
56 views

Estimated marginal trends after regression of isometric log ratio transformed compositional data

I have longitudinal compositional data with four base response variables. I would like to compare the slopes of groups across time, but not for the ilr-transformed composition. I wish to compare the ...
Bryan's user avatar
  • 1,109
3 votes
1 answer
39 views

multiplicity-adjustment w/ unequal variance & pre-planned vs post hoc

The (only) answer to this question says If the variability of the data in each category of your factor is non-constant, that would invalidate ... your multiplicity-adjusted post-hoc p-values. Why ...
Quinn Culver's user avatar
1 vote
0 answers
30 views

When assessing significance of identification variables (e.g., race) after applying a regression model, is it correct to use a Bonferroni adjustment?

I am building a regression model to assess whether being a member of any particular race (compared to the control group, which is White) is associated with a statistically significant loan application ...
mai's user avatar
  • 807
0 votes
0 answers
36 views

Appropriate way to correct for multiple hypotheses with multiple one-way ANOVAs and Tukey's post-hoc

I have a dataset with four groups and 14 different dependent variables. Based on the answer given here, I think the best approach for my analytical question is to perform multiple one-way ANOVAs, ...
Geoff's user avatar
  • 11
0 votes
1 answer
16 views

Is there a statistical test where you test whether any of multiple separate analyses of variances or chi-square tests are statistically significant?

Is there a statistical test where you test whether any of multiple separate analyses of variances or chi-square tests are statistically significant?
jkj's user avatar
  • 1
0 votes
0 answers
27 views

Is correction for multiple comparisons necessary in this instance (single variable)?

Coming here from a more analytical chemistry background, but I figured I would run my current thoughts here before committing to anything after looking around for a bit. I'm used to always using ...
toasty53's user avatar
0 votes
0 answers
30 views

False Discovery Rate-Adjusted Multiple Confidence Intervals after p-value-adjustment

I apologize in advance if my question is incomprehensible, I am very new to this topic. My data consists of 6 dependent variables (DV1-6) and 5 independent variables (IV1-5) of interest. For reasons ...
Joe55711's user avatar
1 vote
0 answers
22 views

is it acceptable to do pairewise comparisons for mixed regressions that aren't independent?

I have a model specified as lmer(log(dv) ~ chosen*tasktype +(1|id)+(1|qs), data) The key issue in the data is that the groups are not indepedent, i.e., ...
Jinjin's user avatar
  • 111
0 votes
0 answers
15 views

Mixing MLE and Hypothesis Testing

In my field of biology, we have spatial data with known distances, so we know a general formula for our error structure that we can use in a generalized least squares context, it is $$ (1-\lambda)\...
A Friendly Fish's user avatar
1 vote
0 answers
71 views

Statistical significance of hierarchical (multi-level) associations

Association rule mining is a common technique in Data mining used to find itemsets which appear together very frequently in large transaction databases (e.g. in Table 3 we see that in a transactions ...
gis_grad_student's user avatar
0 votes
0 answers
10 views

Methods for discriminating between Markov kernels

I'm interested in problems of the following form, which I've deliberately specified a bit vaguely. I would like to know if problems of this kind have been studied, and if so what they're called and ...
N. Virgo's user avatar
  • 425
6 votes
1 answer
182 views

How to adjust p-values in a multiple regression when variables have been a selected by Lasso?

I have designed an analysis where I am testing a lot of variables together. So I first apply a Lasso regression to select the top variables, and then I run a standard (unregularized) multiple ...
PlasmaBinturong's user avatar
1 vote
1 answer
133 views

multiple testing adjustment for fisher exact test

I have 4 groups (G1, G2, G3, G4) consisting of samples of biological material, and we test them in order to identify are there any bacteria present. We also know that all of the samples contain virus ...
ariadnaoliver's user avatar
1 vote
1 answer
22 views

Group Comparison with serial mediation

I'm trying to compare two groups for a serial mediation model. I wrote my code like in this link to make it clearer, here is my code: ...
Cavdar_A's user avatar
2 votes
1 answer
94 views

Am I using emmeans correctly to look at pairwise differences after running a type III ANOVA model?

I am looking at how the weight of female individuals is influenced by the two factors A (5 levels) and B (2 levels) using a type III ANOVA as follows: ...
Insect_biologist's user avatar
2 votes
0 answers
39 views

Sequential analysis - adjusting effect size

I am aware that sequential testing is risky, but it makes sense for me to do it anyway. So, I would like to read up, to do it properly. My idea (which is probably not new): In my reality, the true ...
W_vH's user avatar
  • 339
0 votes
0 answers
27 views

Type I error control for multiple comparisons

In statistics, there is approach, aka, fixed-sequence hirarchical approach to control the Type I error when it comes to multiple tests for different variables. For example, there are 3 tests to be ...
East Liu's user avatar
0 votes
0 answers
21 views

How to do ANOVA/post-hoc analysis when 'control' is used to normalise the data?

I am analysing the transportation rate for a number of different mutants in a particular transportation system. I want to know how I can perform statistical tests on this data in a valid way, given ...
Rachel Jones's user avatar
1 vote
0 answers
10 views

Pairwise trend comparisons for a joint model

I have built a joint model for longitudinal and survival data using the jointModel function in JM. I would like to perform pairwise comparisons of three trends of the longitudinal outcome of the joint ...
Bryan's user avatar
  • 1,109
0 votes
0 answers
28 views

What really happens if I run Tukey without running ANOVA first?

I see that the theory says that if I want to test for difference in mean between 3+ groups I should use ANOVA first, and if ANOVA rejects the null hypothesis then I might want to look into Tukey as a ...
rusiano's user avatar
  • 566

1
2 3 4 5
38