Skip to main content

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

How to report few-shot accuracy for LLMs?

I am comparing three prompting techniques in LLMs to check which one is best. All prompting strategies include three examples for in-context learning (few-shot only, no fine-tuning). If I do greedy ...
Jader Martins's user avatar
4 votes
2 answers
284 views

Is it multiple comparisons?

Our study hypothesis is that metabolic syndrome (MetS) has impact on some outcome. There are at least five definitions of MetS in the literature. One of colleagues suggested using all five definitions ...
 Ivan.S's user avatar
0 votes
0 answers
12 views

How to conduct statistical analysis on scattered paired data [closed]

my data is structured as follows: I have longitudinal paired data of multiple individuals (N = >100 ) for 5 timepoints. But I do not have data for all timepoints for most of them mostly only have ...
CleDie's user avatar
  • 1
1 vote
1 answer
30 views

Mixed Effects with repeated measures design and covariate included

I have a dataset that consists of the oxygen consumption (MO2) values for 3 different species of fish. My experiment was a repeated measures design where I measured MO2 of the same 14 fish per species ...
Allison's user avatar
  • 21
3 votes
1 answer
29 views

Significant results from ANOVA post-hoc Tukey's, insignificant with welch-ANOVA post-hoc Dunnett's T3

I performed one-way ANOVA tests with post-hoc Tukey's to look at multiple comparisons and got highly significant results, but noticed the $F$-value on the ANOVA was high. The SDs in the groups were ...
Harry's user avatar
  • 31
1 vote
1 answer
57 views

Is it possible to do statistic analysis from only knowing my groups's means and sample sizes?

I am writing about a problem my colleagues (in human health field) consider impossible to solve, but I want to believe mathematics have an answer. I am being told it is impossible to do statistical ...
Melissa Gaultier's user avatar
3 votes
1 answer
35 views

Identical SE values for different groups using emtrends

I'm using emtrends to extract slopes and do pairwise comparisons between the groups of my independent variable, controlled for the other variables in the model. ...
Mike's user avatar
  • 131
0 votes
1 answer
24 views

Which statistical test should I use for 3 independent variables and one dependent variable?

What are the tests that I can use for 3 independent variables (ordinal - Likert) and 1 dependent variable (college grades - general average)? I am trying to see if the IV really affects their GWA (DV) ...
Emma's user avatar
  • 3
0 votes
0 answers
23 views

Why does the Dunn's test in online calculator rejects the null hypothesis while my manual computations cannot reject it?

In my own manual calculations of the Dunn's Test in google sheet, using the formula from a YouTube video, I managed to calculate the same Test Statistic as the online calculator. However, using the ...
FifthRevelathor's user avatar
1 vote
1 answer
41 views

How do I do the p-adjustement in statistical tests?

context: a study about which type of bread and/or filling attracts ants the most I have a table with 3 columns : the number of ants attracted, type of bread, type of filling. The question is : do ...
Moez Daly's user avatar
2 votes
2 answers
50 views

type I error in multiple comparisons summary statistics

I understand type I error can increase if we run 'multiple comparisons'. But does that refer to comparisons within a multi-category variable (or a group variable)? or multiple analyses using the same ...
aqen's user avatar
  • 191
0 votes
0 answers
19 views

Multiple comparison tests when removing the intercept?

I have a set of 13 different dependent variables, all tested against 9 independent variables. Here's a simplified example of the tests I'm running using R syntax: ...
millie0725's user avatar
0 votes
1 answer
25 views

validity of multiple comparisons

Suppose I have 2 conditions, A and B, and I estimate 10 different metrics in each condition. If I want to test whether each metric is differentiated across the two conditions, do I have to control for ...
Paris's user avatar
  • 21
1 vote
1 answer
12 views

Combining Z scores of Correlated Test Results

I have the Z scores of a population of N for M different tests, where N is O(10000) and M is ~50. The results of these test are not independent, but we can produce a reasonable estimate of the ...
AspiringStatistician's user avatar
1 vote
1 answer
51 views

Estimated marginal trends after regression of isometric log ratio transformed compositional data

I have longitudinal compositional data with four base response variables. I would like to compare the slopes of groups across time, but not for the ilr-transformed composition. I wish to compare the ...
Bryan's user avatar
  • 1,099
1 vote
1 answer
30 views

multiplicity-adjustment w/ unequal variance & pre-planned vs post hoc

The (only) answer to this question says If the variability of the data in each category of your factor is non-constant, that would invalidate ... your multiplicity-adjusted post-hoc p-values. Why ...
Quinn Culver's user avatar
1 vote
0 answers
29 views

When assessing significance of identification variables (e.g., race) after applying a regression model, is it correct to use a Bonferroni adjustment?

I am building a regression model to assess whether being a member of any particular race (compared to the control group, which is White) is associated with a statistically significant loan application ...
mai's user avatar
  • 807
0 votes
0 answers
33 views

Appropriate way to correct for multiple hypotheses with multiple one-way ANOVAs and Tukey's post-hoc

I have a dataset with four groups and 14 different dependent variables. Based on the answer given here, I think the best approach for my analytical question is to perform multiple one-way ANOVAs, ...
Geoff's user avatar
  • 11
0 votes
1 answer
16 views

Is there a statistical test where you test whether any of multiple separate analyses of variances or chi-square tests are statistically significant?

Is there a statistical test where you test whether any of multiple separate analyses of variances or chi-square tests are statistically significant?
jkj's user avatar
  • 1
0 votes
0 answers
22 views

Is correction for multiple comparisons necessary in this instance (single variable)?

Coming here from a more analytical chemistry background, but I figured I would run my current thoughts here before committing to anything after looking around for a bit. I'm used to always using ...
toasty53's user avatar
0 votes
0 answers
27 views

False Discovery Rate-Adjusted Multiple Confidence Intervals after p-value-adjustment

I apologize in advance if my question is incomprehensible, I am very new to this topic. My data consists of 6 dependent variables (DV1-6) and 5 independent variables (IV1-5) of interest. For reasons ...
Joe55711's user avatar
1 vote
0 answers
22 views

is it acceptable to do pairewise comparisons for mixed regressions that aren't independent?

I have a model specified as lmer(log(dv) ~ chosen*tasktype +(1|id)+(1|qs), data) The key issue in the data is that the groups are not indepedent, i.e., ...
Jinjin's user avatar
  • 111
0 votes
0 answers
12 views

Mixing MLE and Hypothesis Testing

In my field of biology, we have spatial data with known distances, so we know a general formula for our error structure that we can use in a generalized least squares context, it is $$ (1-\lambda)\...
A Friendly Fish's user avatar
1 vote
0 answers
70 views

Statistical significance of hierarchical (multi-level) associations

Association rule mining is a common technique in Data mining used to find itemsets which appear together very frequently in large transaction databases (e.g. in Table 3 we see that in a transactions ...
gis_grad_student's user avatar
0 votes
0 answers
8 views

Methods for discriminating between Markov kernels

I'm interested in problems of the following form, which I've deliberately specified a bit vaguely. I would like to know if problems of this kind have been studied, and if so what they're called and ...
N. Virgo's user avatar
  • 426
6 votes
1 answer
175 views

How to adjust p-values in a multiple regression when variables have been a selected by Lasso?

I have designed an analysis where I am testing a lot of variables together. So I first apply a Lasso regression to select the top variables, and then I run a standard (unregularized) multiple ...
PlasmaBinturong's user avatar
1 vote
1 answer
105 views

multiple testing adjustment for fisher exact test

I have 4 groups (G1, G2, G3, G4) consisting of samples of biological material, and we test them in order to identify are there any bacteria present. We also know that all of the samples contain virus ...
ariadnaoliver's user avatar
1 vote
1 answer
21 views

Group Comparison with serial mediation

I'm trying to compare two groups for a serial mediation model. I wrote my code like in this link to make it clearer, here is my code: ...
Cavdar_A's user avatar
2 votes
1 answer
80 views

Am I using emmeans correctly to look at pairwise differences after running a type III ANOVA model?

I am looking at how the weight of female individuals is influenced by the two factors A (5 levels) and B (2 levels) using a type III ANOVA as follows: ...
Insect_biologist's user avatar
2 votes
0 answers
36 views

Sequential analysis - adjusting effect size

I am aware that sequential testing is risky, but it makes sense for me to do it anyway. So, I would like to read up, to do it properly. My idea (which is probably not new): In my reality, the true ...
W_vH's user avatar
  • 321
0 votes
0 answers
18 views

Type I error control for multiple comparisons

In statistics, there is approach, aka, fixed-sequence hirarchical approach to control the Type I error when it comes to multiple tests for different variables. For example, there are 3 tests to be ...
East Liu's user avatar
0 votes
0 answers
20 views

How to do ANOVA/post-hoc analysis when 'control' is used to normalise the data?

I am analysing the transportation rate for a number of different mutants in a particular transportation system. I want to know how I can perform statistical tests on this data in a valid way, given ...
Rachel Jones's user avatar
1 vote
0 answers
9 views

Pairwise trend comparisons for a joint model

I have built a joint model for longitudinal and survival data using the jointModel function in JM. I would like to perform pairwise comparisons of three trends of the longitudinal outcome of the joint ...
Bryan's user avatar
  • 1,099
0 votes
0 answers
25 views

What really happens if I run Tukey without running ANOVA first?

I see that the theory says that if I want to test for difference in mean between 3+ groups I should use ANOVA first, and if ANOVA rejects the null hypothesis then I might want to look into Tukey as a ...
rusiano's user avatar
  • 566
2 votes
1 answer
35 views

Is it possible to test if the response difference over two time points is significantly different between groups when there are no repeated subjects?

I currently have a dataset where I've collected data between two groups (disease versus control). The experimental design is to evaluate how the response variable changes over time. Thus, we have 4 ...
Syuma's user avatar
  • 125
0 votes
0 answers
14 views

When can we compute Westfall and Young's $p$-values?

Westfall and Young provide a method for adjusting $p$-values following multiple hypothesis tests. I have two questions about when this method can be applied: Can the corrected $p$-values be computed ...
afreelunch's user avatar
0 votes
0 answers
28 views

Statistical methods for comparing performance of ML algorithms on unevenly distributed datasets

Having multiple ML algorithms for outlier/anomaly detection and even more datasets with always two classes (which are intentionally distributed unevenly), I did cross-validation tests for each model ...
tomsko's user avatar
  • 1
2 votes
1 answer
28 views

Should I normalise my dependent variable by a measured variable that I want to be constant?

This is a question about the validity of applying the results of a statistical test, and unfortunately will require some explanation of the experiment in question. Essentially, I want to measure how ...
Rachel Jones's user avatar
0 votes
0 answers
33 views

Is it valid to use unsupervised clustering to assign patients into larger groups before comparing them to controls?

I have very high-dimensional data (lipidomics) with two categories: patient/control and genetic mutation. These mutations have similar phenotypes, but they are considered different diseases. Their ...
maglorismyspiritanimal's user avatar
0 votes
0 answers
32 views

When to use an adjustment method for Kruskal-Wallis + Dunn's and which method?

I need to do multiple pairwise comparisons among 5 groups (Treatment 1, Treatment 2, Treatment 3, Positive Control, Negative Control) for different tests and times. Each group has $10$ individuals, ...
Antonella's user avatar
1 vote
0 answers
32 views

Possible to estimate random effects of a two-level factor? [closed]

I want to conduct a two-way ANOVA with a mixed effect model using lme4. The trial detail is Factor A with 3 levels and Factor B with two levels (3x2) replicated on ...
Workneh's user avatar
  • 11
4 votes
2 answers
356 views

multiple linear mixed models - multiple comparisons adjustment

I recently started to work on a case-control study using repeated measures over time (Modeling repeated measures data in R - Interpretation and Validation). I have several dependent variables and I ...
Ed9012's user avatar
  • 391
0 votes
0 answers
27 views

What does 'Exchangeability' mean in multiple testing?

In A practical guide to methods controlling false discoveries in computational biology, the author mentioned 'Exchangeability': According to Wiki, the definition of 'Exchangeability' is : ...
doraemon's user avatar
  • 208
0 votes
0 answers
24 views

Can bootstrap adjustment for $p$-value in AR model be applied to distributed lag model?

In Wang "Multiple testing correction in time series: rolling window analysis with applications of GWAS methods" (2022), the author mentioned that bootstrap minimum $p$-value can be applied ...
doraemon's user avatar
  • 208
0 votes
0 answers
10 views

Comparing two datasets of communication flows

I know the question of how to compare two datasets has been asked multiple times and I have read a lot about the different tests available but I'm a little lost on what exactly the tests show and what ...
Milan D's user avatar
  • 101
0 votes
0 answers
31 views

Multiple-comparison p-value adjustment in meta-analysis

Do we need to adjust $p$-values for subgroup analyses in a meta-analysis? I'm wondering if the typical subgroup analysis methods, like the ones in packages such as ...
tomas.rosa's user avatar
0 votes
1 answer
16 views

Is there a test or function to perform multiple comparisons for a big dataset?

I have a big dataset of 2304 values : 384 different genes expression values in triplicates and for 2 different groups (ie, different treatments). I only want to compare the same genes between the 2 ...
user avatar
2 votes
1 answer
62 views

Quantitatively distinguishing between multiple-comparison tests

As someone relatively new to applied statistics, I have been trying to better understand some of the best practices for applying statistical methods. I have been recently trying to understand when to ...
Argon's user avatar
  • 153
0 votes
0 answers
17 views

Multiple comparisons (i.e., cluster 1 vs cluster 2+3), how to adjust p-value

I am analyzing the differences in mean of a continuos variable between 3 clusters. I used anova (after testing whether it has a parametric distribution) and obtained p-value lower than 0.05. As post-...
user406121's user avatar
1 vote
0 answers
27 views

Assumptions and validity of post-hoc pairwise comparisons of GLMs

How does packages for pairwise comparisons such as multcomp's glht(mcp) and emmeans pairs() maintain validity in cases of ...
arebru's user avatar
  • 11

1
2 3 4 5
38