Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

461 questions
96 views

Are there realistic/relevant use-cases for one way ANOVA?

I have trouble imagining realistic research questions that can be answered by a one way ANOVA. Two or more ways with interaction terms and perhaps mixed factors (within and between subjects), sure you ...
282 views

Confidence interval for the biggest difference between means

I have a number of samples $X_1,\ldots,X_n$ from Gaussian distributions centered at $\mu_1,\ldots,\mu_n$ respectively. I need to construct a confidence interval for the biggest difference between ...
122 views

Comparing multiple frequency distributions from data and from simulation

I am attempting to analyze the shuffling in the card based online game Magic Arena, which I suspect has a specific bug that causes the distribution of outputs to be biased. Is the approach I have ...
125 views

Do results from the lme-function require adjustment of p-values?

If I run a linear mixed model with the lme() function and get results like these (comparing the score of 4 treatment groups against a placebo group): ...
359 views

How to correct for multiple comparisons in time series data?

I have two time series (length N) and I did a t-test to look at the difference between the two time series time point by time point. So I get N p-values from this. Obviously I need to correct for ...
466 views

Combining Z-scores using Stouffer's approach

Quick background: I require to obtain a single 'score' for comparison between experimental and simulated data. The parameter being tested is the height of various parts (num = 5) of the model. For the ...
187 views

multiple Kruskal-Wallis instead of ANOVA for non-normally distributed data

Apologies in advanced if I supply too little or vague information, since I am a complete newbie in stats and using this forum. So I did a study which evaluated psychopathic personality traits, ...
816 views

Controlling FWER using minP/maxT methods

A common approach for controlling the FWER in genomics analysis is the so-called minP/maxT procedure. See, for example: Dudoit, S., Shaffer, J.P., and Boldrick, J.C. (2003). Multiple Hypothesis ...
2k views

How to best approach ANOVA with multiple response variables that might be correlated and Bonferroni Correction

I need some help verifying a simple analysis. I have completed a simple two factor experiment of more or less equal sample sizes. Let's assume for the case of this question that the sample sizes are ...
194 views

Separate ANOVAs on subsets of data

I often read that, after obtaining complex (e.g., four-way) interactions in a factorial ANOVA, researchers decide to split their data by a factor (e.g., gender) and run separate ANOVAs for these two ...
122 views

Comparing models of different types of ANOVA - multiple comparisons?

Statistical semi-novice question: Just following up this question, the fact that the three types of ANOVA exist with an unbalanced design, and that (presumably) you don't necessarily know a priori ...
1k views

lmPerm p-values and multiple testing

I've started using lmPerm in order to perform regressions in R. The equation I want to fit has the form: ...
22 views

Compare slopes between two groups of equal size

I want to compare how does a dependent variable (total flow) changes with a covariate (total rain) between two groups (stream A & B). Usually, i would go with ANCOVA. But these groups have the ...
89 views

Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...
99 views

A few questions regarding the practice of heterogeneous treatment effect analysis (a.k.a, interaction detection or subgroup analysis) methods

Imagine I am looking at a randomized experiment between a control and one or more treatment conditions. For example, I have a treatment that aims to get people out of debt. I randomize people to ...
23 views

Advice for conducting the same experiment several times?

I'm very new to Bayesian statistics, but I've found myself in a situation where they might be able to help. I'll be running the same experiment repeatedly and analyzing the data every few months. The ...
83 views

35 views

Which test to use to detect if a process has significantly more errors than another?

A have a set of machines running a particular software. I want to do something that's called "canary deployment". This consists of deploying the new version of the software to a subset of machines ...
21 views

Are p-value corrections needed for separate tests with various grouping variables?

I have a dataset (find an example below) in which there are many factor variables (group_a, group_b, ...
38 views

Which test to use on partially overlapping groups

I have data in 4 categories, but they are not completely independent. To explain the data: I have looked at 40 Intelligent Tutoring Systems (ITSs) and determined which approach they use (model tracing,...
244 views

GAM - Smooth comparison with factor variables

I try to compare the splines for my 6 categories. To this end, first I tried the ordered factor approach and posted the result here. This comparison is not sufficient for me since I would like to have ...
42 views

Compare means between months of consecutive years

I have sales data of 150 sellers of an Internet website. For each seller I have daily sales for two years, that is, 730 measures, and there are quite a few days in which the sales are 0. On the other ...
80 views

FDR and the the Benjamini-Hochberg Method

I am trying to understand the Benjamini-Hochberg Method for controlling the false discovery rate. Mathematically, if we are given with m hypothesis testing procedures, we sort the P-Values and reject ...
239 views

Interpreting odds ratio of multiple comparisons from a logistic regression model (using R)

I recently ran a logistic regression on categorical data and ran a Tukey multiple comparisons post hoc analysis using the glht ...
46 views

Does ordering the comparisons by p-value make sense?

In bioinformatics I usually do enrichment test (read hypergeometric test) with many comparisons over the same list. After correcting for multiple comparisons I usually order the results by p-value (...
114 views

Should I use post-hoc tukey HSD or some other test for pairwise comparisons of a factor on a zero-inflated negative binomial mixed model (ZINB)?

I ran a zero-inflated negative binomial mixed model (ZINB) and now have a statistically significant factor (recording type, either NOCA, BCCH, or Human). To assess the differences between these three ...
54 views

Which is the best? Correct interpretation of paiwise comparison results

Consider the situation in which one should investigate, which treatment or treatments (of A, B, C, D, E, F, G, H) is/are the most effective (highest decrease) and which is/are the worst (lowest ...
257 views