Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
5 views

two sample t-test for placebo adjusted values

I have a set of studies (ex: s1....s6) in which each has two arms, a treatment and a placebo. I would like to test for non-inferiority through a matrix of two-sample t-tests w/o accounting for ...
0
votes
0answers
10 views

Multiple comparisons, should I run a one-way multivariate analysis or can I use Mann Whitney?

I am analysing some study data and am unsure the appropriate statistical test to use. The study involves watching a film. 50% of participants saw version 1, 50% saw version 2 - (the difference in ...
0
votes
0answers
11 views

False Discovery Rate (FDR) one-stage and two-stage method

Which is the difference between the classical one-stage and the two-stage method? which one should I report?
-2
votes
0answers
31 views

In hypothesis testing are we not being explicitly partial? [on hold]

Semper necessitas probandi incubit ei qui agit! https://en.wikipedia.org/wiki/Occam%27s_razor Stipulates that science should be based on ignorance the lack of belief. https://en.wikipedia.org/wiki/...
0
votes
1answer
15 views

Comparing one group to many others

I have a quasi-experiment where I'm comparing one experimental group (A) to many control groups (B1, B2, B3, B4, B5). Since this is a quazi-experiment, I do not know for sure that my control groups ...
0
votes
1answer
28 views

what is alternative/opposite to piecemeal statistical approach?

I have only basic knowledge about bio-statistics. For not normally distributed data, I used Kruskal-Wallis test to investigate the statistical significance between different variables. I performed an ...
0
votes
0answers
18 views

A minimum False Discovery Rate of 0.8 when doing multiple hypothesis testing on Logistic Regression Coefficient p-values?

I currently have 8,000 predictor variables and a binary indicator response variable. I then did a logistic regression on each of the predictor variables, i.e., response regressed on predictor variable ...
0
votes
0answers
5 views

Differences in means of raster files

I am analysing long-term data on a raster. I group this data in three parts depending on a factor value. Now, I am taking means in each group pixel by pixel and would like to detect pixels on the map ...
0
votes
0answers
14 views

How to calculate the significance of a pairwise comparison distribution?

I have a variable ($X$) that depends on two parameters (say $P$ and $Q$), each of which can take six values each ($P_1\cdots P_6$ and $Q_1\cdots Q_6$). I take pairwise values of each parameter like $...
0
votes
0answers
4 views

Can I do multiple ANOVAs without correcting for multiple comparisons?

In my study I have one repeated measures factor that I manipulated on two levels and one between-subjects factor that I manipulated on 3 levels (three groups), resulting in a 2x3 design. I also have ...
0
votes
0answers
8 views

GLM 3 factors, significant two way interaction. Pairwaise comparisons?

I used generalized linear models using 3 factors Each factor has 2 levels. My main effects are not significant, nor is the three way interaction But there is a significant interaction between factor ...
2
votes
0answers
43 views

Multiple comparisons correction for dependent comparisons

In this blog post the authors discuss simultaneously estimating quantiles, and constructing a simultaneous confidence envelope for the estimation which covers the whole quantile function. They do this ...
0
votes
0answers
30 views

Type 1 Error correction for multiple comparisons: ANOVA vs multiple regression

After looking into Type 1 error correction protocols for multiple comparisons in ANOVA vs. MR, I found two (conflicting?) messages. The same Type 1 Error corrections that apply to comparisons in ...
0
votes
0answers
20 views

Comparing one group with others when repeated measures with non-parametric tests corrected for multiple comparison

I have a reference method $M_{ref}$ that I would like to compare with other methods $M_1$, $M_2$, $M_3$ when taking measurements (denoted scores from now on) from a data sample of size $N (N > 30)$....
0
votes
0answers
12 views

R: Statistically testing NMDS community differences

Situation I am comparing species communities at 6 different sites. Communities at each site were measured repeatedly over 5 years. I used NMDS plots, based on bray-curtis distance matrices using ...
0
votes
0answers
31 views

How to report the results of the Tukey HSD syntactically/formally correct?

For the ANOVA i use this form: F(2, 1279) = 6.15, p = .002, w^2 = .010. Let's say i did some post hoc testing and the result is this: ...
0
votes
0answers
17 views

Hypothesis testing on set of comparisons experiment

Imagine I'm running an experiment where I'm comparing my method with $M-1$ competitors and I'm collecting the results for $T$ different runs. I would like to test the hypothesis of "my method is the ...
0
votes
0answers
11 views

Grouping hypotheses, and applying FDR to each group separately?

Suppose I have two sets of hypotheses, $A$ and $B$, and I want to control FDR. When is it valid to control the FDR of $A$ and $B$ separately, vs controlling the FDR of $A \cup B$ jointly? One ...
1
vote
1answer
22 views

Steps to conduct ANCOVA between two groups that has 2 covariate?

Using R, I want to compare two groups (1 & 2), each group having two covariate. More specifically, i need to: Within each group, test for significant relationships between the dependent variable (...
0
votes
0answers
10 views

Scheffe's S Method - help with finding the absolute value of l

I am trying to understand Scheffe's S Method and how the absolute value of l is found. In the example I am reading, the absolute value of l is |-.836| but I have absolutely no idea how this value was ...
0
votes
0answers
10 views

Location level a/b testing sample size

We are setting up experiments to measure impact on the city level. Customer in certain cities are treated and we have list of holdout cities with similar behavior. Measurement of lift of the ...
0
votes
0answers
22 views

One-tailed Conover-Iman test?

I'm new to non-parametric statistics and want to do a post hoc analysis of some data after a Kruskal-Wallis test. I am using the implementation of the Conover-Iman test available in the scikit-...
0
votes
0answers
37 views

Kruskal Wallis test or Friedman for multiple preselected unmatched pairs

I am unsure whether I should use Kruskal Wallis test or Friedman's test in the following case, where I am required to use a non-parametric test while compensating for multiple comparison with a post ...
2
votes
0answers
13 views

Multiple testing correction or not?

I have a set of 70 individuals and 90 continuous variables measured for those individuals, and I run univariate models to test for the association to an outcome variable. Is it necessary to always ...
2
votes
1answer
19 views

Best Practice: Multiple Correlations with the same Variable

I am facing the issue that I want to test the correlation between one variable and multiple other variables. The data stems from an Experiment, where I have a control variable of which I want to test ...
0
votes
1answer
20 views

Difference between two dependent variable with two covariates

I have two groups (A & B). First step In each group, I am trying to find the relationships between dependent variable (flow) and 2 covariates (total rain and preflow). Should I: Do a multiple ...
3
votes
0answers
22 views

Compare slopes between two groups of equal size

I want to compare how does a dependent variable (total flow) changes with a covariate (total rain) between two groups (stream A & B). Usually, i would go with ANCOVA. But these groups have the ...
0
votes
0answers
45 views

Is there an implementation of the minP or Westfall & Young corrections?

Let us consider variables $X_1,\dots,X_k$. They are highly dependent. We want to perform some test for any pair $(X_i,Y)$ and imply proper multiple-comparisons correction. As i understand, Westfall &...
1
vote
1answer
35 views

p-value adjustment for correlation matrix

Suppose I have a large number of features, and I want to figure out which of them (if any) are "significantly" correlated. In short, how I can estimate significance in a way that appropriately ...
2
votes
0answers
12 views

Simple effects analysis - correct univariate results for multiple comparisons?

I tested the effect of two factors with both each three levels on a DV. When analyzing my data, the ANOVA revealed two significant main effects and a significant 3x3 interaction which I now plan to ...
1
vote
1answer
49 views

Is it necessary to do a Bonferroni correction on a) exploratory analysis b) correlations or c) if there are 3 different dependent variables

My study is a 2x2 within design and I have a series of 8 hypotheses. These include one ANOVA , one t-test and 4 Pearson correlations. However, the ANOVA and 3 of the correlations include the same ...
1
vote
0answers
29 views

How to compare growth rates estimated from GLMM?

I've counted aphid populations under 5 different treatments for about 10 weeks. There were 4 replicates of every treatment. I'm interested in comparing the first 4-5 weeks, where the populations ...
0
votes
1answer
44 views

How to conclude that all group means are different?

Suppose I have groups $G_1$ to $G_n$ with sample means of the variable of interest $m_1$ to $m_n$ respectively and sample variances $s_1^2$ to $s_n^2$. What tests can I perform to conclude that the ...
0
votes
0answers
20 views

Post hoc for wilcoxon multiple tests

I have a database of brain MRI acquisitions of 20 subjects at two time points. I have elaborated the signal with multiple models. For each model, I have estimated 4-6 parameters maps. At the moment I ...
0
votes
0answers
31 views

Multiple comparisons correction in between-subject classification analysis

I am working with a dataset of 46 subjects (23 controls and 23 patients) fmri brain t-maps each with 4 experimental conditions. The main purpose of my analysis is to find the highest classification ...
1
vote
0answers
22 views

Correction for the number of factors in multi-way ANOVA

QUESTION: Should we do multiple comparisons correction for the number of effects (main effects and interactions) in multi-way ANOVA? I might have failed to find a relevant question, but mostly ...
1
vote
0answers
17 views

How to look at individual test result after we adjust p value for multiple comparison

For example, we have 100 biomarkers. We are making 100 t test for biomaker measures over two treatments. After p-value adjustments, we find 10 of them are significant. Now we want to look into these ...
0
votes
1answer
18 views

Post hoc analysis after two-way repeated measures ANOVA

I am measuring test performance at 10 different timepoints, before and after drug application. So far I ran two-way repeated measures ANOVA and determined there is a significant timepoint, drug, and ...
1
vote
1answer
37 views

Logical Hypothesis Modification

I performed an experiment with a treatment and a control group. It just so happened that my treatment could be modified to either increase or decrease my dependent variable. So, I decided to test two ...
0
votes
1answer
91 views

Setting up contrasts in lmer?

0 I have 139 subjects (ID), with measurements taken at two time points (Time1, Time2), at 148 brain regions, a dependent measure called volume, and a covariate called thickness. Each subject has 148 ...
1
vote
0answers
28 views

What test should one use when dealing with different treatments in plant clones?

In an experiment involving plant clones there are: 20+ treatments, 5 technical replicates for each treatment (plant clones in which the treatments were tested). Variable of interest is Biomass. Due ...
0
votes
1answer
24 views

Should I adjust for family-wise error when doing many logistic regressions and t tests

In my biomarker analysis, there are 100 biomarkers. Each biomarker is measured at 4 different time points. I'm looking for any biomarker that is associated with outcome variable at any timepoints. ...
0
votes
2answers
30 views

Paired t-test with inequality in $H_0$

I have read that paired $t$-test works assuming a null hypothesis with equality, i.e: $$H_0 : \mu = \mu_0 $$ Is there any way to do the same test with an inequality, such as: $$H_0 : \mu \geq \mu_0 $$ ...
0
votes
0answers
22 views

Statistical test to compare the mean of two samples

There's a competition with 20 categories, and each category has 3 winners. I have an array of the winner's performance of the next year (60 values), and other array with the average of the competitors ...
0
votes
0answers
17 views

Using Multiple Comparisons for Kruskal Wallis Follow up analysis

I am performing the Kruskal Wallis test on ~100 samples and wish to group statistically ‘similar’ samples together. The method I am using is described in Andy Field’s ...
1
vote
4answers
35 views

How to combine multiple dose levels in a study for meta-analysis

I am conducting a meta-analysis of RCTs for treatment A vs. control. Some of the studies have multiple dose levels of A. Should I combine these groups with different doses of A and compute one effect ...
2
votes
1answer
50 views

Is it correct to run a mixed model (2-ways anova) on a unbalanced and incomplete design?

I am studying the effect of land use on biomass. I am struggling to find the right model to correctly analyse my data. Data description I have surveyed land use changes in 20 fields for 3 years. I ...
0
votes
0answers
11 views

Multi variable clustering cartographical data

Firstly , my knowledge in statistics is very limited , so excuse me if I'm ask a none well placed question. I have a country ,( example : USA ) , and i have 3 set of data 1) position (lat , lng ) ...
0
votes
1answer
25 views

Is this a valid method to adjust for multiple testing?

Assuming I have 100 variables and I do a t-test for each of them without adjusting the p-values, and get 8 significant p-values out of the 100 tests. If my alpha is 5% can I just say that since I ...
2
votes
0answers
36 views

Alternative hypotheses when conducting pairwise comparisons

When conducting pairwise comparisons, what restrictions are there on the alternative hypotheses? That is to say, is it okay if some tests are two tailed while others are one tailed?