Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1
vote
0answers
13 views

Correction for the number of factors in multi-way ANOVA

QUESTION: Should we do multiple comparisons correction for the number of effects (main effects and interactions) in multi-way ANOVA? I might have failed to find a relevant question, but mostly ...
1
vote
0answers
15 views

How to look at individual test result after we adjust p value for multiple comparison

For example, we have 100 biomarkers. We are making 100 t test for biomaker measures over two treatments. After p-value adjustments, we find 10 of them are significant. Now we want to look into these ...
0
votes
1answer
12 views

Post hoc analysis after two-way repeated measures ANOVA

I am measuring test performance at 10 different timepoints, before and after drug application. So far I ran two-way repeated measures ANOVA and determined there is a significant timepoint, drug, and ...
1
vote
1answer
30 views

Logical Hypothesis Modification

I performed an experiment with a treatment and a control group. It just so happened that my treatment could be modified to either increase or decrease my dependent variable. So, I decided to test two ...
1
vote
1answer
75 views

Setting up contrasts in lmer?

0 I have 139 subjects (ID), with measurements taken at two time points (Time1, Time2), at 148 brain regions, a dependent measure called volume, and a covariate called thickness. Each subject has 148 ...
1
vote
0answers
27 views

What test should one use when dealing with different treatments in plant clones?

In an experiment involving plant clones there are: 20+ treatments, 5 technical replicates for each treatment (plant clones in which the treatments were tested). Variable of interest is Biomass. Due ...
0
votes
1answer
19 views

Should I adjust for family-wise error when doing many logistic regressions and t tests

In my biomarker analysis, there are 100 biomarkers. Each biomarker is measured at 4 different time points. I'm looking for any biomarker that is associated with outcome variable at any timepoints. ...
0
votes
2answers
26 views

Paired t-test with inequality in $H_0$

I have read that paired $t$-test works assuming a null hypothesis with equality, i.e: $$H_0 : \mu = \mu_0 $$ Is there any way to do the same test with an inequality, such as: $$H_0 : \mu \geq \mu_0 $$ ...
0
votes
0answers
22 views

Statistical test to compare the mean of two samples

There's a competition with 20 categories, and each category has 3 winners. I have an array of the winner's performance of the next year (60 values), and other array with the average of the competitors ...
0
votes
0answers
12 views

Using Multiple Comparisons for Kruskal Wallis Follow up analysis

I am performing the Kruskal Wallis test on ~100 samples and wish to group statistically ‘similar’ samples together. The method I am using is described in Andy Field’s ...
1
vote
4answers
28 views

How to combine multiple dose levels in a study for meta-analysis

I am conducting a meta-analysis of RCTs for treatment A vs. control. Some of the studies have multiple dose levels of A. Should I combine these groups with different doses of A and compute one effect ...
2
votes
1answer
40 views

Is it correct to run a mixed model (2-ways anova) on a unbalanced and incomplete design?

I am studying the effect of land use on biomass. I am struggling to find the right model to correctly analyse my data. Data description I have surveyed land use changes in 20 fields for 3 years. I ...
0
votes
0answers
10 views

Multi variable clustering cartographical data

Firstly , my knowledge in statistics is very limited , so excuse me if I'm ask a none well placed question. I have a country ,( example : USA ) , and i have 3 set of data 1) position (lat , lng ) ...
0
votes
1answer
25 views

Is this a valid method to adjust for multiple testing?

Assuming I have 100 variables and I do a t-test for each of them without adjusting the p-values, and get 8 significant p-values out of the 100 tests. If my alpha is 5% can I just say that since I ...
2
votes
0answers
34 views

Alternative hypotheses when conducting pairwise comparisons

When conducting pairwise comparisons, what restrictions are there on the alternative hypotheses? That is to say, is it okay if some tests are two tailed while others are one tailed?
0
votes
0answers
10 views

Reference line for QQPlots: robust regression vs quartiles method

Using the car package, I drew two qqPlots of 144 p values from correlated tests on one dataset. The top plot uses a reference line drawn by "quartiles" and the bottom plot uses a line drawn by "robust"...
1
vote
0answers
19 views

A conceptual question about multiple comparison

tl;dr: I feel like multiple comparison can be easily manipulated to have significant results. As far as I understand, the concept of multiple comparison problem is very simple. As we are conducting ...
0
votes
0answers
12 views

Test for comparing 4 groups, 18 conditions and 1 numerical variable

I hope I'll manage to make my problem understandable. I've got a data set which contains 4 groups (CG, TG, HG, IG) across 18 conditions (e.g. 18 verbal cues) and 1 numerical variable (the participant'...
1
vote
1answer
32 views

What does a p value histogram that is “normally” distributed mean?

Let's say I performed 100 tests and and want to correct for multiple comparisons. Before I do so, I plot the unaltered p values in a histogram to see what the distribution looks like. If the null ...
0
votes
1answer
27 views

Test to compare on an ordinal scale, two groups of data that have covariates

I have two (Test and Control) groups of data. Data in each group is divided into 10 (ordinal) period. In each of these periods, number of observation ranges anywhere from 22-53. The assumption is that ...
0
votes
0answers
28 views

Softmax regression vector features?

I'm working on a difficult classification problem where there are a very large number of known classes (~50k). I have only about 20k labelled points, but these only represent <1% of the possible ...
5
votes
2answers
149 views

What is the definition of dataset (for Bonferroni purposes)?

I'm having difficulties finding a clear rule for when a series of test should be considered a multiple comparison and when we should apply p-value corrections (like Bonferroni). I understand ...
1
vote
0answers
18 views

Multivariate $t$ ('mvt') - adjustment in emmeans in R

I am doing post-hoc comparisons of contrasts based on linear mixed models I built in R. I am using the emmeans package for the comparisons. One of the default ...
2
votes
1answer
68 views

Hypothesis Test of Hypothesis Tests

I am doing a linguistics project in which I am comparing the % of the time "um" follows a contraction vs. the % of time "um" follows what I call the full form (e.g. full form of don't is do not). I ...
0
votes
1answer
24 views

How to compensate for different statistical power in multiple comparisons? [closed]

Imagine we are performing 10 comparisons on a single variable of choice. The 10 respective experiments have different statistical powers. Now imagine that the least powerful and the most powerful ...
0
votes
1answer
47 views

Generalization of one-sample t-test for multiple groups

If I have some data*, and want to test the hypothesis that it has a given mean $\mu_0$, I know of the possibility of using a one-sample t-test. Here, $\mu_0=0$ could be the null-hypothesis of a ...
0
votes
1answer
12 views

Appropriate Multiple Testing Correction for Correlation Matrix

I have $N$ samples of $D$ variables, for which I want to study the correlation. Specifically, I want to know if any of these variables are correlated. I can get estimates and a matrix of p-values in R ...
1
vote
0answers
21 views

Pairwise non-parametric test?

I have 5 groups of patients (different sizes), for which I measured some property that is continuous but not normally distributed. I would like to compare every pair of groups in respect to this ...
0
votes
1answer
36 views

Correcting for multiple post hoc tests in factorial ANOVA

I have done a two-way repeated-measures ANOVA with factor A having 3 levels and factor B having 5 levels. I am doing post hoc comparisons between all 3 levels of factor A within each of the 5 levels ...
1
vote
3answers
62 views

Why not remove an experimental group and do a t-Test instead of an ANOVA?

This has never been made clear to me before, so I would love some help. Lets say I have 3 experimental groups of animals (A,B,C) A is the baseline control, B is treatment x, and C is treatment y. I ...
1
vote
0answers
56 views

Pairwise comparisons after ANOVA with Tukey and Bonferroni methods in r

The built-in functions TukeyHSD and LSD.test from R perform pairwise comparisons. In order to maintain the type I error, the significance level $0.05$ is divided by the number of comparisons $k$. Do ...
1
vote
0answers
22 views

FIsher's LSD test is appropriate for restricted mean comparisons

Suppose that we perform two way ANOVA with interactions, with factors $A$ and $B$ with three level each $A_{1},A_{2},A_{3}$ and $B_{1},B_{2},B_{3}$. We know that Fisher's LSD test is used for ...
2
votes
1answer
26 views

Repeated measures models multiple testing adjustment

Perhaps I am missing something obvious here, but I can't find an answer to this. Let's say we run 10 individual repeated measures models with factor(time) as fixed effect with 5 separate timepoints ...
0
votes
0answers
11 views

Are differences in proportions meaningful in a sample's segmentation

I am setting up an A/B test of a webpage where users will be randomly distributed between either A or B. I'll be using the N-1 Proportion Test to determine which page converts better than the other. ...
0
votes
1answer
27 views

Unsure which data to use for ANOVA test

I have determined that a one-way ANOVA test of variances will work best for my data. However, the nature of my data makes me very concerned about making any claims based on significance or not. My ...
2
votes
0answers
32 views

Model selection using p-values - tree inference

Suppose I have some i.i.d. normal observations from $\mathbb{R}^f$ with parameters $(\mu, \Sigma)$ and $\Sigma$ is known to be the identity matrix. I have the following hypotheses: $H_0^i$: $\mu_i = ...
0
votes
0answers
9 views

Most suitable comparison between groups

I have a neural network based algorithm which it was trained with 5 different datasets (A, B, C, D and E). For each one, 10 training runs was made {(A1, A2, ..., A10), (B1, ..., B10), ...}. Then, it ...
0
votes
1answer
25 views

Best way to compare several cases to a null model

At the moment I am doing analysis in some data that is (currently) divided by number of individuals in a group. More or less like: Groups w/ two individuals: Value_1, Value_2, ..., Value_N Groups w/ ...
4
votes
3answers
123 views

Fairness of 6 sided die NOT using Chi Square Test

So I was reading a lot of online articles using the chi-square test to determine the fairness of a die. Correct me if I am wrong, but isn't the drawbacks of using a goodness of fit test is that you ...
0
votes
0answers
14 views

Sample size calculation for multiple hypotheses testing on two or more metrics

Let's say I want to run an experiment where there's a control and there's a feature group (say, 50-50 random split). The treatment can affect the feature group in 1, 2 or three ways (or any ...
0
votes
1answer
33 views

Post-hoc corrections for multiple t-tests

I am comparing two groups (young vs. old adults) for differences in 5 brain regions (size) with independent t-tests. I also compare performance on three different aspects of a task with independent t-...
0
votes
0answers
13 views

Correct group of factorial and repeated-measure ANOVA

I'm currently having trouble defining what type of ANOVA I need to run as well as how to run it correctly. This is the following setup: I have 4 groups of 30 people doing a 2-session task (one after ...
0
votes
0answers
23 views

Pairwise comparisons for a nested effects

This is basically a theoretical question which is based on the following R output. I have the following R output which contains both crossed and nested effects. The group(3 levels) and phase(2 levels)...
0
votes
0answers
18 views

Any AI/ML approaches for analysis of competing hypotheses?

I know there are significant development about sentiment analysis in text e.g. "x is awful", "x is bad" or "y is fantastic". But are there similar approaches to compare hypotheses? e.g. given a ...
1
vote
1answer
74 views

Correcting for multiple pairwise comparisons with GAM objects {mgcv} in R

My question concerns pairwise comparisons of factor levels in a gam object. I have a dataframe, df, containing reaction time data (in ms) to stimuli varying in ...
-1
votes
0answers
10 views

Combining p-values for parwise tests (single cell data)

I'm working on single-cell data. To find genes which are markers of groups, I perform pairwise comparison between groups of cells. I was used to take the maximal FDR as a criterium to decide if a ...
-1
votes
0answers
11 views

Permutation and multiple testing - permuting the test labels instead of sample labels?

I'm comparing $m$ p-values, where $m$ is very large. Every p-value comes from a two-sample $t$-test of two groups of size $n_1$ and $n_2$, where the $n_i$ are very small ($2$ or $3$). This leads to a $...
-1
votes
0answers
16 views

How to calculate false discovery rate for this situation

Let's say I have run 20 mediation models using structural equation modeling and have found evidence for significant effects of interest (i.e., mediational/indirect paths) in 5 out of 20 models. These ...
2
votes
0answers
34 views

Which test to use to detect if a process has significantly more errors than another?

A have a set of machines running a particular software. I want to do something that's called "canary deployment". This consists of deploying the new version of the software to a subset of machines ...
1
vote
1answer
45 views

Adjusting p-values for multiple ANOVAs

I am performing multiple (around 40) multi-way ANOVAs on a dataset, one for each outcome variable. There are three factors involved: cohort, pre/post, and tissue type. For each outcome variable, I ...