Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1
vote
0answers
13 views

Correction for the number of factors in multi-way ANOVA

QUESTION: Should we do multiple comparisons correction for the number of effects (main effects and interactions) in multi-way ANOVA? I might have failed to find a relevant question, but mostly ...
1
vote
0answers
15 views

How to look at individual test result after we adjust p value for multiple comparison

For example, we have 100 biomarkers. We are making 100 t test for biomaker measures over two treatments. After p-value adjustments, we find 10 of them are significant. Now we want to look into these ...
9
votes
3answers
8k views

How to correctly apply the Nemenyi post-hoc test after the Friedman test

I'm comparing the performance of multiple algorithms on multiple data sets. Since those performance measurements are not guaranteed to be normally distributed, I chose the Friedman Test with the ...
1
vote
1answer
30 views

Logical Hypothesis Modification

I performed an experiment with a treatment and a control group. It just so happened that my treatment could be modified to either increase or decrease my dependent variable. So, I decided to test two ...
1
vote
1answer
75 views

Setting up contrasts in lmer?

0 I have 139 subjects (ID), with measurements taken at two time points (Time1, Time2), at 148 brain regions, a dependent measure called volume, and a covariate called thickness. Each subject has 148 ...
0
votes
1answer
12 views

Post hoc analysis after two-way repeated measures ANOVA

I am measuring test performance at 10 different timepoints, before and after drug application. So far I ran two-way repeated measures ANOVA and determined there is a significant timepoint, drug, and ...
0
votes
1answer
336 views

Correcting for multiple comparisons after multiple ANOVAs

I've had to run 5 different ANOVAs (identical 4-way mixed design ANOVAs including the same factors, but on a different dependent variable [a gait parameter] each time). In this case, what would be the ...
1
vote
0answers
27 views

What test should one use when dealing with different treatments in plant clones?

In an experiment involving plant clones there are: 20+ treatments, 5 technical replicates for each treatment (plant clones in which the treatments were tested). Variable of interest is Biomass. Due ...
0
votes
1answer
19 views

Should I adjust for family-wise error when doing many logistic regressions and t tests

In my biomarker analysis, there are 100 biomarkers. Each biomarker is measured at 4 different time points. I'm looking for any biomarker that is associated with outcome variable at any timepoints. ...
0
votes
2answers
26 views

Paired t-test with inequality in $H_0$

I have read that paired $t$-test works assuming a null hypothesis with equality, i.e: $$H_0 : \mu = \mu_0 $$ Is there any way to do the same test with an inequality, such as: $$H_0 : \mu \geq \mu_0 $$ ...
1
vote
1answer
329 views

Sample Size for Multiple Testing

I want to calculate sample size for a classic multiple testing problem of multiple doses in clinical trials. I have 3 groups: "high dose", "low dose" and "placebo", and I wish to test if at least one ...
0
votes
0answers
22 views

Statistical test to compare the mean of two samples

There's a competition with 20 categories, and each category has 3 winners. I have an array of the winner's performance of the next year (60 values), and other array with the average of the competitors ...
1
vote
4answers
28 views

How to combine multiple dose levels in a study for meta-analysis

I am conducting a meta-analysis of RCTs for treatment A vs. control. Some of the studies have multiple dose levels of A. Should I combine these groups with different doses of A and compute one effect ...
2
votes
1answer
40 views

Is it correct to run a mixed model (2-ways anova) on a unbalanced and incomplete design?

I am studying the effect of land use on biomass. I am struggling to find the right model to correctly analyse my data. Data description I have surveyed land use changes in 20 fields for 3 years. I ...
0
votes
0answers
12 views

Using Multiple Comparisons for Kruskal Wallis Follow up analysis

I am performing the Kruskal Wallis test on ~100 samples and wish to group statistically ‘similar’ samples together. The method I am using is described in Andy Field’s ...
0
votes
0answers
10 views

Multi variable clustering cartographical data

Firstly , my knowledge in statistics is very limited , so excuse me if I'm ask a none well placed question. I have a country ,( example : USA ) , and i have 3 set of data 1) position (lat , lng ) ...
0
votes
1answer
25 views

Is this a valid method to adjust for multiple testing?

Assuming I have 100 variables and I do a t-test for each of them without adjusting the p-values, and get 8 significant p-values out of the 100 tests. If my alpha is 5% can I just say that since I ...
2
votes
0answers
34 views

Alternative hypotheses when conducting pairwise comparisons

When conducting pairwise comparisons, what restrictions are there on the alternative hypotheses? That is to say, is it okay if some tests are two tailed while others are one tailed?
1
vote
1answer
459 views

Paired t-test for 3 groups

I have three paired groups, let's call them A, B and C. I want to check if A's mean is statistically better than B and if A's mean is better than C. I don't care how B and C interact. I believe I ...
1
vote
3answers
62 views

Why not remove an experimental group and do a t-Test instead of an ANOVA?

This has never been made clear to me before, so I would love some help. Lets say I have 3 experimental groups of animals (A,B,C) A is the baseline control, B is treatment x, and C is treatment y. I ...
0
votes
0answers
10 views

Reference line for QQPlots: robust regression vs quartiles method

Using the car package, I drew two qqPlots of 144 p values from correlated tests on one dataset. The top plot uses a reference line drawn by "quartiles" and the bottom plot uses a line drawn by "robust"...
4
votes
1answer
493 views

False Coverage Rate for confidence intervals of positive likelihood ratios of multiple dependent tests

I want to calculate confidence intervals of positive likelihood ratios of multiple dependent tests. To adjust the confidence level for the 'problem of multiple comparisons', I believe I should ...
5
votes
1answer
2k views

Comparing observed and predicted values across several measurements

As a neuropsychology graduate student with some experience in statistics (I'm usually the guy other psychologists come to with statistics problems after trying it themselves but before seeing a ...
2
votes
0answers
360 views

Comparing facial curvature of two human faces

I need help determining if what I'm doing is sufficient. I have extracted the "curvature" from two human faces (essentially the curve that goes from one ear, down the face to the chin, and back up to ...
11
votes
1answer
703 views

How to compare of two Spearman correlations matrices?

I have two non-parametric rank correlations matrices emp and sim (for example, based on Spearman's $\rho$ rank correlation ...
1
vote
1answer
638 views

Correcting for multiple 2-way anova testing

I want to study the effect of two factors on a paramater that I measure in several regions. I'm running a 2-way ANOVA test for each region (about a dozen) and I am wondering how I can deal to avoid ...
1
vote
0answers
19 views

A conceptual question about multiple comparison

tl;dr: I feel like multiple comparison can be easily manipulated to have significant results. As far as I understand, the concept of multiple comparison problem is very simple. As we are conducting ...
0
votes
0answers
28 views

Softmax regression vector features?

I'm working on a difficult classification problem where there are a very large number of known classes (~50k). I have only about 20k labelled points, but these only represent <1% of the possible ...
0
votes
1answer
27 views

Test to compare on an ordinal scale, two groups of data that have covariates

I have two (Test and Control) groups of data. Data in each group is divided into 10 (ordinal) period. In each of these periods, number of observation ranges anywhere from 22-53. The assumption is that ...
0
votes
0answers
12 views

Test for comparing 4 groups, 18 conditions and 1 numerical variable

I hope I'll manage to make my problem understandable. I've got a data set which contains 4 groups (CG, TG, HG, IG) across 18 conditions (e.g. 18 verbal cues) and 1 numerical variable (the participant'...
1
vote
1answer
32 views

What does a p value histogram that is “normally” distributed mean?

Let's say I performed 100 tests and and want to correct for multiple comparisons. Before I do so, I plot the unaltered p values in a histogram to see what the distribution looks like. If the null ...
2
votes
1answer
513 views

How to interpret grouping result in Tukey's HSD test

Assuming there are three methods to kill bugs, and we want to know whether their performance differs. The Tukey's HSD test gives the following grouping results: ...
0
votes
1answer
459 views

What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
5
votes
2answers
149 views

What is the definition of dataset (for Bonferroni purposes)?

I'm having difficulties finding a clear rule for when a series of test should be considered a multiple comparison and when we should apply p-value corrections (like Bonferroni). I understand ...
2
votes
1answer
68 views

Hypothesis Test of Hypothesis Tests

I am doing a linguistics project in which I am comparing the % of the time "um" follows a contraction vs. the % of time "um" follows what I call the full form (e.g. full form of don't is do not). I ...
1
vote
0answers
18 views

Multivariate $t$ ('mvt') - adjustment in emmeans in R

I am doing post-hoc comparisons of contrasts based on linear mixed models I built in R. I am using the emmeans package for the comparisons. One of the default ...
3
votes
1answer
359 views

Is Uplift Modeling a solution to Multiple Comparisons problem?

If you want to identify a particular user segment for whom an experiment produced some lift or incremental effect over the control treatment, wouldn’t it be more direct to do uplift modeling? Uplift ...
0
votes
1answer
24 views

How to compensate for different statistical power in multiple comparisons? [closed]

Imagine we are performing 10 comparisons on a single variable of choice. The 10 respective experiments have different statistical powers. Now imagine that the least powerful and the most powerful ...
0
votes
1answer
47 views

Generalization of one-sample t-test for multiple groups

If I have some data*, and want to test the hypothesis that it has a given mean $\mu_0$, I know of the possibility of using a one-sample t-test. Here, $\mu_0=0$ could be the null-hypothesis of a ...
1
vote
0answers
21 views

Pairwise non-parametric test?

I have 5 groups of patients (different sizes), for which I measured some property that is continuous but not normally distributed. I would like to compare every pair of groups in respect to this ...
2
votes
1answer
729 views

Performance testing: detecting changes in distributions over time

I have an interesting application I’d like some advice on. A common task when working with stochastic systems in an engineering context is testing for regressions and improvements in performance. ...
0
votes
1answer
12 views

Appropriate Multiple Testing Correction for Correlation Matrix

I have $N$ samples of $D$ variables, for which I want to study the correlation. Specifically, I want to know if any of these variables are correlated. I can get estimates and a matrix of p-values in R ...
1
vote
1answer
528 views

Overall goodness-of-fit/p-value for multiple items IRT model in R (ltm)

The data set which I am trying to analyse is Student Test Data. I have a data of responses (either 1/correct response or 0/incorrect response) on some questions of a set of students. I have fitted a "...
0
votes
1answer
342 views

How to use the Scheffe correction

I have 4 conditional correlation matrices $C_1, \ldots, C_4$. I want to test each entry in the $k$th matrix, say $\hat{r}_{ij}^{(k)}$, against a reference value $r_{ij}^{0}$ value in a matrix $C_0$. ...
0
votes
1answer
27 views

Unsure which data to use for ANOVA test

I have determined that a one-way ANOVA test of variances will work best for my data. However, the nature of my data makes me very concerned about making any claims based on significance or not. My ...
0
votes
1answer
36 views

Correcting for multiple post hoc tests in factorial ANOVA

I have done a two-way repeated-measures ANOVA with factor A having 3 levels and factor B having 5 levels. I am doing post hoc comparisons between all 3 levels of factor A within each of the 5 levels ...
1
vote
0answers
56 views

Pairwise comparisons after ANOVA with Tukey and Bonferroni methods in r

The built-in functions TukeyHSD and LSD.test from R perform pairwise comparisons. In order to maintain the type I error, the significance level $0.05$ is divided by the number of comparisons $k$. Do ...
1
vote
0answers
22 views

FIsher's LSD test is appropriate for restricted mean comparisons

Suppose that we perform two way ANOVA with interactions, with factors $A$ and $B$ with three level each $A_{1},A_{2},A_{3}$ and $B_{1},B_{2},B_{3}$. We know that Fisher's LSD test is used for ...
2
votes
1answer
26 views

Repeated measures models multiple testing adjustment

Perhaps I am missing something obvious here, but I can't find an answer to this. Let's say we run 10 individual repeated measures models with factor(time) as fixed effect with 5 separate timepoints ...
0
votes
0answers
11 views

Are differences in proportions meaningful in a sample's segmentation

I am setting up an A/B test of a webpage where users will be randomly distributed between either A or B. I'll be using the N-1 Proportion Test to determine which page converts better than the other. ...