Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

6
votes
0answers
96 views

Are there realistic/relevant use-cases for one way ANOVA?

I have trouble imagining realistic research questions that can be answered by a one way ANOVA. Two or more ways with interaction terms and perhaps mixed factors (within and between subjects), sure you ...
5
votes
0answers
282 views

Confidence interval for the biggest difference between means

I have a number of samples $X_1,\ldots,X_n $ from Gaussian distributions centered at $\mu_1,\ldots,\mu_n $ respectively. I need to construct a confidence interval for the biggest difference between ...
4
votes
0answers
122 views

Comparing multiple frequency distributions from data and from simulation

I am attempting to analyze the shuffling in the card based online game Magic Arena, which I suspect has a specific bug that causes the distribution of outputs to be biased. Is the approach I have ...
4
votes
0answers
125 views

Do results from the lme-function require adjustment of p-values?

If I run a linear mixed model with the lme() function and get results like these (comparing the score of 4 treatment groups against a placebo group): ...
4
votes
0answers
359 views

How to correct for multiple comparisons in time series data?

I have two time series (length N) and I did a t-test to look at the difference between the two time series time point by time point. So I get N p-values from this. Obviously I need to correct for ...
4
votes
0answers
466 views

Combining Z-scores using Stouffer's approach

Quick background: I require to obtain a single 'score' for comparison between experimental and simulated data. The parameter being tested is the height of various parts (num = 5) of the model. For the ...
4
votes
0answers
187 views

multiple Kruskal-Wallis instead of ANOVA for non-normally distributed data

Apologies in advanced if I supply too little or vague information, since I am a complete newbie in stats and using this forum. So I did a study which evaluated psychopathic personality traits, ...
4
votes
0answers
816 views

Controlling FWER using minP/maxT methods

A common approach for controlling the FWER in genomics analysis is the so-called minP/maxT procedure. See, for example: Dudoit, S., Shaffer, J.P., and Boldrick, J.C. (2003). Multiple Hypothesis ...
4
votes
0answers
2k views

How to best approach ANOVA with multiple response variables that might be correlated and Bonferroni Correction

I need some help verifying a simple analysis. I have completed a simple two factor experiment of more or less equal sample sizes. Let's assume for the case of this question that the sample sizes are ...
4
votes
0answers
194 views

Separate ANOVAs on subsets of data

I often read that, after obtaining complex (e.g., four-way) interactions in a factorial ANOVA, researchers decide to split their data by a factor (e.g., gender) and run separate ANOVAs for these two ...
4
votes
0answers
122 views

Comparing models of different types of ANOVA - multiple comparisons?

Statistical semi-novice question: Just following up this question, the fact that the three types of ANOVA exist with an unbalanced design, and that (presumably) you don't necessarily know a priori ...
4
votes
0answers
1k views

lmPerm p-values and multiple testing

I've started using lmPerm in order to perform regressions in R. The equation I want to fit has the form: ...
3
votes
0answers
22 views

Compare slopes between two groups of equal size

I want to compare how does a dependent variable (total flow) changes with a covariate (total rain) between two groups (stream A & B). Usually, i would go with ANCOVA. But these groups have the ...
3
votes
0answers
89 views

Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...
3
votes
0answers
99 views

A few questions regarding the practice of heterogeneous treatment effect analysis (a.k.a, interaction detection or subgroup analysis) methods

Imagine I am looking at a randomized experiment between a control and one or more treatment conditions. For example, I have a treatment that aims to get people out of debt. I randomize people to ...
3
votes
0answers
23 views

Advice for conducting the same experiment several times?

I'm very new to Bayesian statistics, but I've found myself in a situation where they might be able to help. I'll be running the same experiment repeatedly and analyzing the data every few months. The ...
3
votes
0answers
83 views

Estimating parameters for a large number of fair and unfair coins

Suppose we have N coins. We know that at least half of them are fair. Suppose we flip coin $i$ for $n_i$ times and $x_i$ denotes number of heads. Given $n = (n_1, \dots , n_N)$ and $x = (x_1,\dots,x_N)...
3
votes
0answers
100 views

Determine Chi Square Group Differences in multiple choice questionnaire

I'm trying to determine if there is a group difference in answering a multiple choice question. For example, I want to answer the question of, do "Change Instigators" (group A) answer "communication"...
3
votes
0answers
270 views

Statistical method to combine/summarize multiple R squared values

I have a set of measurements using two different methods. I Have 6 samples that I have measurements for using both methods. I am trying to demonstrate how comparable the measurements are from the ...
3
votes
0answers
112 views

Non normally distributed data and comparing correlation

I have data from a research that was conducted by me, the data refer to four conditions made of two levels of time (pretest and post-test) and two levels of group (control and intervention). Shapiro-...
3
votes
0answers
74 views

Bayesian association testing and simulation

I read some Bayesian references recently and came up with some basic or naive questions about Bayesian inference. My research question is to test the association between predictors and outcome, and it ...
3
votes
0answers
449 views

p-value distribution and the applicability of different multiple testing correction methods

I came across this issue as I was analysing some genomic data sets looking for differential DNA methylation. I have generated p-values from the comparisons I am making, and am now looking into the ...
3
votes
0answers
563 views

Testing the difference between multiple paired proportions with different sample sizes

Is there a hypothesis test similar to the paired t-test for testing if the difference between a number of paired proportion estimates is significantly different from zero? The data consists of 2 lists ...
3
votes
0answers
442 views

Multiple comparison of non-normal, heteroscedastic data. What test should I use?

I have a set of brain pathology data. These were obtained by counting certain parameters in the brain. Due to availability of human brains, the amount of cases vary a lot across the different groups ...
3
votes
0answers
59 views

The link between FWER and max/min statistic

I am currently looking at lecture slides about fMRI data and I have the following: Let $H_{0_i}$ be the hypothesis that there is no activation in voxel $i$ (a cube area in the brain, a 3d version of ...
3
votes
0answers
51 views

Multiple comparisons of parameters from non-linear regression

Hei, I want to compare parameters a_i and b_i estimated by nonlinear regression (y=a_i*x/(b_i+x)) for different data-sets (let's say 8 different data sets). I have calculated the non-linear ...
3
votes
0answers
74 views

Running sequential tests as data is collected - practical advice for researchers

I've observed that many experimental labs in my field don't do a power analysis before they start collecting data. In practice, it seems that many researchers collect data and periodically run ...
3
votes
0answers
152 views

Regression analysis to estimate affinity or antipathy among celebrity wedding parties

Please consider the following plot of a graph: Each node represents a wedding party while each edge is the relation between the two parties expressed in terms of the number of people who attended ...
3
votes
0answers
176 views

Comparing the mean of several exponentially distributed variables

(Possibly related to Is there a nonparametric equivalent of Tukey HSD?) Given a set of several exponentially distributed variables (representing data which is modeled this way), I'd like to check ...
3
votes
0answers
63 views

Detecting subset of high sensitivity corresponding to genotype

Background There is a data set from a clinical trial, involving about 1000 patients received an behavior intervention. The intervention is supposed to affect a variable x1, and the change in x1 ...
3
votes
0answers
243 views

Alpha inflation in the context of usability studies

In a hypothetical usability experiment, two interface changes are tested against a baseline. The interface changes are cumulative, i.e. variant B is an extension of A. We measure usability according ...
3
votes
0answers
253 views

How to draw a Hasse diagram of a large number of pairwise comparisons?

I have 100 pairwise comparisons where my results show that 33% of the pairwise comparison tests are non-significant. How can I draw draw a Hasse diagram to represent this? Are there any tips to ...
3
votes
0answers
1k views

What are the possible multiple comparison methods for paired data?

I have been working with simulations, among them scheduling simulations. Each simulation chooses the number of steps, the arrival rate of new jobs and the duration of each step for each job. Thus, ...
3
votes
0answers
204 views

How to determine multivariate confidence in laymans terms

I am trying to determine colors to best grab a web-users attention. I have a simple system which randomly chooses one of 4 foreground colors, and one of 6 background colors. I then save the stats in ...
2
votes
0answers
16 views

Multiple testing correction or not?

I have a set of 70 individuals and 90 continuous variables measured for those individuals, and I run univariate models to test for the association to an outcome variable. Is it necessary to always ...
2
votes
0answers
37 views

Alternative hypotheses when conducting pairwise comparisons

When conducting pairwise comparisons, what restrictions are there on the alternative hypotheses? That is to say, is it okay if some tests are two tailed while others are one tailed?
2
votes
0answers
36 views

Model selection using p-values - tree inference

Suppose I have some i.i.d. normal observations from $\mathbb{R}^f$ with parameters $(\mu, \Sigma)$ and $\Sigma$ is known to be the identity matrix. I have the following hypotheses: $H_0^i$: $\mu_i = ...
2
votes
0answers
35 views

Which test to use to detect if a process has significantly more errors than another?

A have a set of machines running a particular software. I want to do something that's called "canary deployment". This consists of deploying the new version of the software to a subset of machines ...
2
votes
0answers
21 views

Are p-value corrections needed for separate tests with various grouping variables?

I have a dataset (find an example below) in which there are many factor variables (group_a, group_b, ...
2
votes
0answers
38 views

Which test to use on partially overlapping groups

I have data in 4 categories, but they are not completely independent. To explain the data: I have looked at 40 Intelligent Tutoring Systems (ITSs) and determined which approach they use (model tracing,...
2
votes
0answers
244 views

GAM - Smooth comparison with factor variables

I try to compare the splines for my 6 categories. To this end, first I tried the ordered factor approach and posted the result here. This comparison is not sufficient for me since I would like to have ...
2
votes
0answers
42 views

Compare means between months of consecutive years

I have sales data of 150 sellers of an Internet website. For each seller I have daily sales for two years, that is, 730 measures, and there are quite a few days in which the sales are 0. On the other ...
2
votes
0answers
80 views

FDR and the the Benjamini-Hochberg Method

I am trying to understand the Benjamini-Hochberg Method for controlling the false discovery rate. Mathematically, if we are given with m hypothesis testing procedures, we sort the P-Values and reject ...
2
votes
0answers
239 views

Interpreting odds ratio of multiple comparisons from a logistic regression model (using R)

I recently ran a logistic regression on categorical data and ran a Tukey multiple comparisons post hoc analysis using the glht ...
2
votes
0answers
46 views

Does ordering the comparisons by p-value make sense?

In bioinformatics I usually do enrichment test (read hypergeometric test) with many comparisons over the same list. After correcting for multiple comparisons I usually order the results by p-value (...
2
votes
0answers
114 views

Should I use post-hoc tukey HSD or some other test for pairwise comparisons of a factor on a zero-inflated negative binomial mixed model (ZINB)?

I ran a zero-inflated negative binomial mixed model (ZINB) and now have a statistically significant factor (recording type, either NOCA, BCCH, or Human). To assess the differences between these three ...
2
votes
0answers
54 views

Which is the best? Correct interpretation of paiwise comparison results

Consider the situation in which one should investigate, which treatment or treatments (of A, B, C, D, E, F, G, H) is/are the most effective (highest decrease) and which is/are the worst (lowest ...
2
votes
0answers
257 views

Do Tukey and Games-Howell tests need additional p value corrections?

I was reading documentation of function posthocTGH (the "Note" section, where authors expressed doubt about additional corrections to p-values) and linked ...
2
votes
0answers
53 views

Fisher Test and corrections for multiple comparisons

When I do 4 comparisons (namely 4 groups of data) using Fisher test, do I have to correct the results for multiple comparisons (therefore reducing the accepted p for significance to everything below 0....
2
votes
0answers
44 views

Trying to detect a correlation across multiple data sets (same stat, different countries)

I have two data points, "A" and "B." A and B are stats the effect countries, so I have data of A and B for four countries, Canada, the UK, the US and Australia. Due to changes in reporting, each ...