Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
11 views

Can't sample all conditions for each participant — strategies to compensate?

I am attempting to design a study using Category Comparison Rating / Double Stimulus Comparison Scale to evaluate participant perception of different stimulus conditions. The original study design ...
2
votes
1answer
30 views

LME/Multiple regression with many predictors and limited DV range

For a single-case patient study (case profile), I have 20 IVs such as medication intake, amount of sleep etc; and one DV representing the severity of the symptom reported by the patient in each of k=...
0
votes
1answer
14 views

How to compare coexpression networks from two conditions with different sample sizes [closed]

I have gene expression data from two conditions having different sample sizes i.e 8 and 32. I want to build a correlation among all genes (for both conditions) for comparison. So, I used "psych" ...
0
votes
0answers
9 views

Multiple regression and multiple comparisons 3 groups

I'm trying to compare the effect of an IV on my DV, where my data are separated into 3 groups. Specifically, in one group there is a significant correlation between IV and DV, but in the other two ...
0
votes
0answers
9 views

Comparing language loss among two types of bilinguals

I have two groups: Simultaneous and Sequential Bilinguals I have three measures: grammatical complexity, lexical complexity, and grammatical accuracy (all are quantified differently) I will conduct ...
0
votes
0answers
4 views

Multiple comparisons in mixed effect models

I am trying to run ~16 mixed effect models in my study design. I have four different dependent variables and want to run models for each one where I progressively add random effects and test each's ...
0
votes
0answers
11 views

Single summary statistic and p-value for a large number of correlated tests

I have several thousand measurements from an experiment, $\rho_i$, which are correlated. I can compute p-values $\Pr(\rho{>}\rho_i | \mathbf {H_0}_i)$ for each measurement by monte-carlo sampling ...
1
vote
1answer
15 views

Multiple comparisons without adjusting

If I am comparing two groups with respect to 20 different variables, and do not adjust for the number of comparisons I make, will my probability of making a type-I error be larger than alpha = 0.05? ...
0
votes
0answers
2 views

Can mantel tests be evaluated on a large stratified matrix or how to combine multiple independent mantel tests for one overall result?

I have 11 pairwise matrices that are all asking the same question. I collected the data at 11 different sites, and what to know if there is an overall relationship (not just at one particular site). ...
0
votes
0answers
10 views

Compare and visualization of multiple output data files

I have 100 data files corresponding to 100 samples. For these 100 data files, I have processed data in 4 different ways to generate result output files. Out of the 4 results files per sample, I ...
0
votes
1answer
37 views

Testing whether the means within group are different from each other (R)

My Y variable is binary and takes the values 0 and 1. The same applies for the independent variable X. I've been able to create the means within the group, here: ...
1
vote
0answers
8 views

Inflation of effect size estimation among significant results

Correction for multiple testing is often performed on the alpha value. However, multiple testing is, I think, likely to inflate also the estimated effect size of the most significant effects found. ...
-1
votes
0answers
12 views

Why do I get different results for my dummy coded multiple regression on categorical variables and Tukey's HSD?

We are testing the effects of multiple compounds on the diffusion of Tau along axons in cells that express different Tau constructs. The study includes 2 factors: 1. Construct (2 levels) 2. Compound (...
0
votes
0answers
24 views

Linear models and multiple comparisons on time-series data

I have conducted a study on subjects corresponding to two groups G1 and G2. Each subject in the group was tested under two conditions C1 and C2. The testing under each condition was done across a ...
0
votes
1answer
42 views

multcomp() vs emmeans() for multiple comparisons

I have two sets of models. The first set contains a number of logistic models, fitted using glm(), with different binary dependent variables. The second set ...
9
votes
2answers
137 views

How to test whether $\mu_1 < \mu_2 <\mu_3$?

Suppose I have three independent groups, with mean $\mu_1,~ \mu_2,~\mu_3$ respectively. How can I test whether $\mu_1 < \mu_2 <\mu_3$ or not using $n_1,~n_2,~n_3$ samples from each group? I ...
0
votes
0answers
14 views

Multivariate survival analysis and multiple comparisons in R using survival package

I'm studying how different light cycles and feeding regiments affect survival times in different species and would like to compute pairwise comparisons between the survival curves of 12 different ...
0
votes
0answers
23 views

False Discovery Rate (FDR) q-value

If tests are not completely independent (as in the case of 30 proteins tested for associations, partly correlated to each other, and measured by a multiplex immunoassay) it is justified to consider ...
4
votes
0answers
121 views

Comparing multiple frequency distributions from data and from simulation

I am attempting to analyze the shuffling in the card based online game Magic Arena, which I suspect has a specific bug that causes the distribution of outputs to be biased. Is the approach I have ...
0
votes
1answer
36 views

Multiple testing (bonferroni?) with GLM

Hi everyone: I scored four mutually-exclusive behaviors (e.g. feeding, brooding, standing, preening) on a number of observations of birds in two different treatments, and two different nesting ...
0
votes
2answers
38 views

Multiple comparison problem

If I conduct 19 tests (e.g. 19 pairwise comparisons of treatment means in an experiment), what is the probability that I will find at least one P value < 0.05 due to chance alone? With number of ...
0
votes
0answers
8 views

Computation Example of the Morisita Index

I have been interested in this index for a while now. I am coming from a biology background. Apparently, given various quantifiable characteristics of two different species, you can compare how ...
1
vote
1answer
79 views

FDR calculation for each p value

I am trying to understand how FDR is calculated and what to understand from it. I selected R as language so the example is in R Lets look at an example of 10 samples ...
0
votes
0answers
10 views

How to analyze EEG time-frequency data in a factorial design

I am trying to find the best strategy to analyze a set of EEG time-frequency data from 24 subjects, using 64 electrodes. This is an exploratory study as I have no a-priori hypothesis on where or when ...
0
votes
0answers
21 views

Applying the Pocock boundary rule when p-value is not set to 0.05

I would like to set my new p-value threshold correcting for both multiple comparisons and interim analyses. In a very easy example with two analyses (1 interim and 1 final) according to the Pocock ...
0
votes
0answers
10 views

two sample t-test for placebo adjusted values

I have a set of studies (ex: s1....s6) in which each has two arms, a treatment and a placebo. I would like to test for non-inferiority through a matrix of two-sample t-tests w/o accounting for ...
0
votes
0answers
12 views

Multiple comparisons, should I run a one-way multivariate analysis or can I use Mann Whitney?

I am analysing some study data and am unsure the appropriate statistical test to use. The study involves watching a film. 50% of participants saw version 1, 50% saw version 2 - (the difference in ...
1
vote
1answer
22 views

Comparing one group to many others

I have a quasi-experiment where I'm comparing one experimental group (A) to many control groups (B1, B2, B3, B4, B5). Since this is a quazi-experiment, I do not know for sure that my control groups ...
0
votes
1answer
30 views

what is alternative/opposite to piecemeal statistical approach?

I have only basic knowledge about bio-statistics. For not normally distributed data, I used Kruskal-Wallis test to investigate the statistical significance between different variables. I performed an ...
0
votes
0answers
20 views

A minimum False Discovery Rate of 0.8 when doing multiple hypothesis testing on Logistic Regression Coefficient p-values?

I currently have 8,000 predictor variables and a binary indicator response variable. I then did a logistic regression on each of the predictor variables, i.e., response regressed on predictor variable ...
0
votes
0answers
7 views

Differences in means of raster files

I am analysing long-term data on a raster. I group this data in three parts depending on a factor value. Now, I am taking means in each group pixel by pixel and would like to detect pixels on the map ...
0
votes
0answers
18 views

How to calculate the significance of a pairwise comparison distribution?

I have a variable ($X$) that depends on two parameters (say $P$ and $Q$), each of which can take six values each ($P_1\cdots P_6$ and $Q_1\cdots Q_6$). I take pairwise values of each parameter like $...
0
votes
0answers
8 views

Can I do multiple ANOVAs without correcting for multiple comparisons?

In my study I have one repeated measures factor that I manipulated on two levels and one between-subjects factor that I manipulated on 3 levels (three groups), resulting in a 2x3 design. I also have ...
0
votes
0answers
13 views

GLM 3 factors, significant two way interaction. Pairwaise comparisons?

I used generalized linear models using 3 factors Each factor has 2 levels. My main effects are not significant, nor is the three way interaction But there is a significant interaction between factor ...
7
votes
1answer
166 views

Multiple comparisons correction for dependent comparisons

In this blog post the authors discuss simultaneously estimating quantiles, and constructing a simultaneous confidence envelope for the estimation which covers the whole quantile function. They do this ...
0
votes
0answers
66 views

Type 1 Error correction for multiple comparisons: ANOVA vs multiple regression

After looking into Type 1 error correction protocols for multiple comparisons in ANOVA vs. MR, I found two (conflicting?) messages. The same Type 1 Error corrections that apply to comparisons in ...
0
votes
0answers
21 views

Comparing one group with others when repeated measures with non-parametric tests corrected for multiple comparison

I have a reference method $M_{ref}$ that I would like to compare with other methods $M_1$, $M_2$, $M_3$ when taking measurements (denoted scores from now on) from a data sample of size $N (N > 30)$....
0
votes
0answers
22 views

R: Statistically testing NMDS community differences

Situation I am comparing species communities at 6 different sites. Communities at each site were measured repeatedly over 5 years. I used NMDS plots, based on bray-curtis distance matrices using ...
0
votes
0answers
119 views

How to report the results of the Tukey HSD syntactically/formally correct?

For the ANOVA i use this form: F(2, 1279) = 6.15, p = .002, w^2 = .010. Let's say i did some post hoc testing and the result is this: ...
0
votes
0answers
18 views

Hypothesis testing on set of comparisons experiment

Imagine I'm running an experiment where I'm comparing my method with $M-1$ competitors and I'm collecting the results for $T$ different runs. I would like to test the hypothesis of "my method is the ...
0
votes
0answers
13 views

Grouping hypotheses, and applying FDR to each group separately?

Suppose I have two sets of hypotheses, $A$ and $B$, and I want to control FDR. When is it valid to control the FDR of $A$ and $B$ separately, vs controlling the FDR of $A \cup B$ jointly? One ...
1
vote
1answer
29 views

Steps to conduct ANCOVA between two groups that has 2 covariate?

Using R, I want to compare two groups (1 & 2), each group having two covariate. More specifically, i need to: Within each group, test for significant relationships between the dependent variable (...
0
votes
0answers
12 views

Scheffe's S Method - help with finding the absolute value of l

I am trying to understand Scheffe's S Method and how the absolute value of l is found. In the example I am reading, the absolute value of l is |-.836| but I have absolutely no idea how this value was ...
0
votes
0answers
10 views

Location level a/b testing sample size

We are setting up experiments to measure impact on the city level. Customer in certain cities are treated and we have list of holdout cities with similar behavior. Measurement of lift of the ...
0
votes
0answers
51 views

One-tailed Conover-Iman test?

I'm new to non-parametric statistics and want to do a post hoc analysis of some data after a Kruskal-Wallis test. I am using the implementation of the Conover-Iman test available in the scikit-...
0
votes
0answers
44 views

Kruskal Wallis test or Friedman for multiple preselected unmatched pairs

I am unsure whether I should use Kruskal Wallis test or Friedman's test in the following case, where I am required to use a non-parametric test while compensating for multiple comparison with a post ...
2
votes
0answers
14 views

Multiple testing correction or not?

I have a set of 70 individuals and 90 continuous variables measured for those individuals, and I run univariate models to test for the association to an outcome variable. Is it necessary to always ...
2
votes
1answer
21 views

Best Practice: Multiple Correlations with the same Variable

I am facing the issue that I want to test the correlation between one variable and multiple other variables. The data stems from an Experiment, where I have a control variable of which I want to test ...
0
votes
1answer
22 views

Difference between two dependent variable with two covariates

I have two groups (A & B). First step In each group, I am trying to find the relationships between dependent variable (flow) and 2 covariates (total rain and preflow). Should I: Do a multiple ...
3
votes
0answers
22 views

Compare slopes between two groups of equal size

I want to compare how does a dependent variable (total flow) changes with a covariate (total rain) between two groups (stream A & B). Usually, i would go with ANCOVA. But these groups have the ...