# Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

461 questions
0answers
96 views

### Are there realistic/relevant use-cases for one way ANOVA?

I have trouble imagining realistic research questions that can be answered by a one way ANOVA. Two or more ways with interaction terms and perhaps mixed factors (within and between subjects), sure you ...
0answers
282 views

### Confidence interval for the biggest difference between means

I have a number of samples $X_1,\ldots,X_n$ from Gaussian distributions centered at $\mu_1,\ldots,\mu_n$ respectively. I need to construct a confidence interval for the biggest difference between ...
0answers
122 views

### Comparing multiple frequency distributions from data and from simulation

I am attempting to analyze the shuffling in the card based online game Magic Arena, which I suspect has a specific bug that causes the distribution of outputs to be biased. Is the approach I have ...
0answers
125 views

### Do results from the lme-function require adjustment of p-values?

If I run a linear mixed model with the lme() function and get results like these (comparing the score of 4 treatment groups against a placebo group): ...
0answers
359 views

### How to correct for multiple comparisons in time series data?

I have two time series (length N) and I did a t-test to look at the difference between the two time series time point by time point. So I get N p-values from this. Obviously I need to correct for ...
0answers
466 views

### Combining Z-scores using Stouffer's approach

Quick background: I require to obtain a single 'score' for comparison between experimental and simulated data. The parameter being tested is the height of various parts (num = 5) of the model. For the ...
0answers
187 views

### multiple Kruskal-Wallis instead of ANOVA for non-normally distributed data

Apologies in advanced if I supply too little or vague information, since I am a complete newbie in stats and using this forum. So I did a study which evaluated psychopathic personality traits, ...
0answers
816 views

### Controlling FWER using minP/maxT methods

A common approach for controlling the FWER in genomics analysis is the so-called minP/maxT procedure. See, for example: Dudoit, S., Shaffer, J.P., and Boldrick, J.C. (2003). Multiple Hypothesis ...
0answers
2k views

### How to best approach ANOVA with multiple response variables that might be correlated and Bonferroni Correction

I need some help verifying a simple analysis. I have completed a simple two factor experiment of more or less equal sample sizes. Let's assume for the case of this question that the sample sizes are ...
0answers
194 views

### Separate ANOVAs on subsets of data

I often read that, after obtaining complex (e.g., four-way) interactions in a factorial ANOVA, researchers decide to split their data by a factor (e.g., gender) and run separate ANOVAs for these two ...
0answers
122 views

### Comparing models of different types of ANOVA - multiple comparisons?

Statistical semi-novice question: Just following up this question, the fact that the three types of ANOVA exist with an unbalanced design, and that (presumably) you don't necessarily know a priori ...
0answers
1k views

### lmPerm p-values and multiple testing

I've started using lmPerm in order to perform regressions in R. The equation I want to fit has the form: ...
0answers
22 views

### Compare slopes between two groups of equal size

I want to compare how does a dependent variable (total flow) changes with a covariate (total rain) between two groups (stream A & B). Usually, i would go with ANCOVA. But these groups have the ...
0answers
89 views

### Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...
0answers
99 views

### A few questions regarding the practice of heterogeneous treatment effect analysis (a.k.a, interaction detection or subgroup analysis) methods

Imagine I am looking at a randomized experiment between a control and one or more treatment conditions. For example, I have a treatment that aims to get people out of debt. I randomize people to ...
0answers
23 views

### Advice for conducting the same experiment several times?

I'm very new to Bayesian statistics, but I've found myself in a situation where they might be able to help. I'll be running the same experiment repeatedly and analyzing the data every few months. The ...
0answers
83 views

0answers
35 views

### Which test to use to detect if a process has significantly more errors than another?

A have a set of machines running a particular software. I want to do something that's called "canary deployment". This consists of deploying the new version of the software to a subset of machines ...
0answers
21 views

### Are p-value corrections needed for separate tests with various grouping variables?

I have a dataset (find an example below) in which there are many factor variables (group_a, group_b, ...
0answers
38 views

### Which test to use on partially overlapping groups

I have data in 4 categories, but they are not completely independent. To explain the data: I have looked at 40 Intelligent Tutoring Systems (ITSs) and determined which approach they use (model tracing,...
0answers
244 views

### GAM - Smooth comparison with factor variables

I try to compare the splines for my 6 categories. To this end, first I tried the ordered factor approach and posted the result here. This comparison is not sufficient for me since I would like to have ...
0answers
42 views

### Compare means between months of consecutive years

I have sales data of 150 sellers of an Internet website. For each seller I have daily sales for two years, that is, 730 measures, and there are quite a few days in which the sales are 0. On the other ...
0answers
80 views

### FDR and the the Benjamini-Hochberg Method

I am trying to understand the Benjamini-Hochberg Method for controlling the false discovery rate. Mathematically, if we are given with m hypothesis testing procedures, we sort the P-Values and reject ...
0answers
239 views

### Interpreting odds ratio of multiple comparisons from a logistic regression model (using R)

I recently ran a logistic regression on categorical data and ran a Tukey multiple comparisons post hoc analysis using the glht ...
0answers
46 views

### Does ordering the comparisons by p-value make sense?

In bioinformatics I usually do enrichment test (read hypergeometric test) with many comparisons over the same list. After correcting for multiple comparisons I usually order the results by p-value (...
0answers
114 views

### Should I use post-hoc tukey HSD or some other test for pairwise comparisons of a factor on a zero-inflated negative binomial mixed model (ZINB)?

I ran a zero-inflated negative binomial mixed model (ZINB) and now have a statistically significant factor (recording type, either NOCA, BCCH, or Human). To assess the differences between these three ...
0answers
54 views

### Which is the best? Correct interpretation of paiwise comparison results

Consider the situation in which one should investigate, which treatment or treatments (of A, B, C, D, E, F, G, H) is/are the most effective (highest decrease) and which is/are the worst (lowest ...
0answers
257 views

### Do Tukey and Games-Howell tests need additional p value corrections?

I was reading documentation of function posthocTGH (the "Note" section, where authors expressed doubt about additional corrections to p-values) and linked ...
0answers
53 views

### Fisher Test and corrections for multiple comparisons

When I do 4 comparisons (namely 4 groups of data) using Fisher test, do I have to correct the results for multiple comparisons (therefore reducing the accepted p for significance to everything below 0....
0answers
44 views

### Trying to detect a correlation across multiple data sets (same stat, different countries)

I have two data points, "A" and "B." A and B are stats the effect countries, so I have data of A and B for four countries, Canada, the UK, the US and Australia. Due to changes in reporting, each ...