Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
4 views

two sample t-test for placebo adjusted values

I have a set of studies (ex: s1....s6) in which each has two arms, a treatment and a placebo. I would like to test for non-inferiority through a matrix of two-sample t-tests w/o accounting for ...
1
vote
1answer
68 views

GLMM summary and multcomp differences

Thanks for taking the time to read my question. I am using R to run a GLMM because I have taken repeated measurements on 100 individuals. The explanatory variable it is a proportion so I am using ...
0
votes
1answer
27 views

what is alternative/opposite to piecemeal statistical approach?

I have only basic knowledge about bio-statistics. For not normally distributed data, I used Kruskal-Wallis test to investigate the statistical significance between different variables. I performed an ...
0
votes
0answers
10 views

Multiple comparisons, should I run a one-way multivariate analysis or can I use Mann Whitney?

I am analysing some study data and am unsure the appropriate statistical test to use. The study involves watching a film. 50% of participants saw version 1, 50% saw version 2 - (the difference in ...
0
votes
0answers
11 views

False Discovery Rate (FDR) one-stage and two-stage method

Which is the difference between the classical one-stage and the two-stage method? which one should I report?
-2
votes
0answers
31 views

In hypothesis testing are we not being explicitly partial? [on hold]

Semper necessitas probandi incubit ei qui agit! https://en.wikipedia.org/wiki/Occam%27s_razor Stipulates that science should be based on ignorance the lack of belief. https://en.wikipedia.org/wiki/...
0
votes
1answer
15 views

Comparing one group to many others

I have a quasi-experiment where I'm comparing one experimental group (A) to many control groups (B1, B2, B3, B4, B5). Since this is a quazi-experiment, I do not know for sure that my control groups ...
0
votes
0answers
18 views

A minimum False Discovery Rate of 0.8 when doing multiple hypothesis testing on Logistic Regression Coefficient p-values?

I currently have 8,000 predictor variables and a binary indicator response variable. I then did a logistic regression on each of the predictor variables, i.e., response regressed on predictor variable ...
0
votes
0answers
5 views

Differences in means of raster files

I am analysing long-term data on a raster. I group this data in three parts depending on a factor value. Now, I am taking means in each group pixel by pixel and would like to detect pixels on the map ...
2
votes
0answers
43 views

Multiple comparisons correction for dependent comparisons

In this blog post the authors discuss simultaneously estimating quantiles, and constructing a simultaneous confidence envelope for the estimation which covers the whole quantile function. They do this ...
0
votes
1answer
201 views

Ordinal Logistic Regression with Polr: Compare multiple treatments

Let's say I have information about three stages of cancer (Stage 1, Stage 2, Stage 3) and three treatment groups. Now I'd like to compare if the odds ratio for all of these three groups are different. ...
0
votes
0answers
788 views

Adjusting p-values of multiple correlations

I do a correlation test in r with cor.test over two matrices, M and A, the final correlation matrix is C. The cor.test provides ...
67
votes
1answer
7k views

40,000 neuroscience papers might be wrong

I saw this article in the Economist about a seemingly devastating paper [1] casting doubt on "something like 40,000 published [fMRI] studies." The error, they say, is because of "erroneous statistical ...
0
votes
0answers
14 views

How to calculate the significance of a pairwise comparison distribution?

I have a variable ($X$) that depends on two parameters (say $P$ and $Q$), each of which can take six values each ($P_1\cdots P_6$ and $Q_1\cdots Q_6$). I take pairwise values of each parameter like $...
0
votes
0answers
4 views

Can I do multiple ANOVAs without correcting for multiple comparisons?

In my study I have one repeated measures factor that I manipulated on two levels and one between-subjects factor that I manipulated on 3 levels (three groups), resulting in a 2x3 design. I also have ...
0
votes
0answers
8 views

GLM 3 factors, significant two way interaction. Pairwaise comparisons?

I used generalized linear models using 3 factors Each factor has 2 levels. My main effects are not significant, nor is the three way interaction But there is a significant interaction between factor ...
0
votes
0answers
20 views

Comparing one group with others when repeated measures with non-parametric tests corrected for multiple comparison

I have a reference method $M_{ref}$ that I would like to compare with other methods $M_1$, $M_2$, $M_3$ when taking measurements (denoted scores from now on) from a data sample of size $N (N > 30)$....
1
vote
1answer
211 views

Right way for permutation test for correlation

I have a data matrix with size of 24 by 369, consisting of 4 classes. I want to evaluate the variable importance using permutation test. I know there are lots of methods to find informative variables ...
0
votes
0answers
30 views

Type 1 Error correction for multiple comparisons: ANOVA vs multiple regression

After looking into Type 1 error correction protocols for multiple comparisons in ANOVA vs. MR, I found two (conflicting?) messages. The same Type 1 Error corrections that apply to comparisons in ...
1
vote
1answer
214 views

Multiple t tests or an ANOVA?

I have low anxiety and high anxiety participants (21 in high anxiety and 20 In low). I measured anxiety and then categorised Pts according to this. They completed two memory tasks. In one memory ...
0
votes
0answers
12 views

R: Statistically testing NMDS community differences

Situation I am comparing species communities at 6 different sites. Communities at each site were measured repeatedly over 5 years. I used NMDS plots, based on bray-curtis distance matrices using ...
0
votes
1answer
213 views

Binary Data with Count Dependent Variable?

Howdy internet strangers! I've put a sample of my data below, but basically I have the number of accidental falls in a particular set of patients vs. drugs that the patients were taking. The drugs ...
1
vote
1answer
105 views

Controlling FDR when p-values are bimodal

I have a quick question about using the Benjamini-Hochberg procedure to control FDR using the p.adjust() function in R. I have read that in order to properly apply ...
0
votes
0answers
17 views

Hypothesis testing on set of comparisons experiment

Imagine I'm running an experiment where I'm comparing my method with $M-1$ competitors and I'm collecting the results for $T$ different runs. I would like to test the hypothesis of "my method is the ...
0
votes
0answers
31 views

How to report the results of the Tukey HSD syntactically/formally correct?

For the ANOVA i use this form: F(2, 1279) = 6.15, p = .002, w^2 = .010. Let's say i did some post hoc testing and the result is this: ...
1
vote
1answer
22 views

Steps to conduct ANCOVA between two groups that has 2 covariate?

Using R, I want to compare two groups (1 & 2), each group having two covariate. More specifically, i need to: Within each group, test for significant relationships between the dependent variable (...
0
votes
0answers
11 views

Grouping hypotheses, and applying FDR to each group separately?

Suppose I have two sets of hypotheses, $A$ and $B$, and I want to control FDR. When is it valid to control the FDR of $A$ and $B$ separately, vs controlling the FDR of $A \cup B$ jointly? One ...
2
votes
1answer
34 views

Repeated measures models multiple testing adjustment

Perhaps I am missing something obvious here, but I can't find an answer to this. Let's say we run 10 individual repeated measures models with factor(time) as fixed effect with 5 separate timepoints ...
0
votes
0answers
10 views

Scheffe's S Method - help with finding the absolute value of l

I am trying to understand Scheffe's S Method and how the absolute value of l is found. In the example I am reading, the absolute value of l is |-.836| but I have absolutely no idea how this value was ...
0
votes
1answer
222 views

compare multiple algorithms in the same dataset

I want to do a comparison between multiple algorithms when they applied to the same dataset. For example, I have a set of texts in Arabic and another version translated to English. I did a sentiment ...
0
votes
0answers
10 views

Location level a/b testing sample size

We are setting up experiments to measure impact on the city level. Customer in certain cities are treated and we have list of holdout cities with similar behavior. Measurement of lift of the ...
8
votes
1answer
3k views

Multiple comparisons with many groups

I would like to determine if using multiple comparisons test would be appropriate for my data. I used the Kruskal-Wallis test to determine if there were differences in mean inhibition among $17$ ...
0
votes
0answers
21 views

One-tailed Conover-Iman test?

I'm new to non-parametric statistics and want to do a post hoc analysis of some data after a Kruskal-Wallis test. I am using the implementation of the Conover-Iman test available in the scikit-...
0
votes
0answers
37 views

Kruskal Wallis test or Friedman for multiple preselected unmatched pairs

I am unsure whether I should use Kruskal Wallis test or Friedman's test in the following case, where I am required to use a non-parametric test while compensating for multiple comparison with a post ...
2
votes
3answers
268 views

Should I adjust the significance levels in a multiple (linear) regression with dummy variables, when making many comparisons?

I have an experiment with a continuous dependent variable, and an independent variable with a number of categories, k. The categories come from a randomised controlled trial I ran. There is 1 control ...
2
votes
0answers
13 views

Multiple testing correction or not?

I have a set of 70 individuals and 90 continuous variables measured for those individuals, and I run univariate models to test for the association to an outcome variable. Is it necessary to always ...
19
votes
3answers
3k views

Confusion with false discovery rate and multiple testing (on Colquhoun 2014)

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can ...
3
votes
2answers
3k views

Can one obtain a p-value for all levels of factor when doing a GLM?

Please help! I have recently been criticized for using pairwise comparisons to explain all three levels of a factor within a negative binomial GLM rather than all levels at once. I was told that it is ...
1
vote
0answers
29 views

How to compare growth rates estimated from GLMM?

I've counted aphid populations under 5 different treatments for about 10 weeks. There were 4 replicates of every treatment. I'm interested in comparing the first 4-5 weeks, where the populations ...
9
votes
1answer
509 views

What's the formula for the Benjamini-Hochberg adjusted p-value?

I understand the procedure and what it controls. So what's the formula for the adjusted p-value in the BH procedure for multiple comparisons? Just now I realized the original BH didn't produce ...
2
votes
1answer
19 views

Best Practice: Multiple Correlations with the same Variable

I am facing the issue that I want to test the correlation between one variable and multiple other variables. The data stems from an Experiment, where I have a control variable of which I want to test ...
0
votes
0answers
45 views

Is there an implementation of the minP or Westfall & Young corrections?

Let us consider variables $X_1,\dots,X_k$. They are highly dependent. We want to perform some test for any pair $(X_i,Y)$ and imply proper multiple-comparisons correction. As i understand, Westfall &...
0
votes
1answer
20 views

Difference between two dependent variable with two covariates

I have two groups (A & B). First step In each group, I am trying to find the relationships between dependent variable (flow) and 2 covariates (total rain and preflow). Should I: Do a multiple ...
1
vote
1answer
35 views

p-value adjustment for correlation matrix

Suppose I have a large number of features, and I want to figure out which of them (if any) are "significantly" correlated. In short, how I can estimate significance in a way that appropriately ...
3
votes
0answers
22 views

Compare slopes between two groups of equal size

I want to compare how does a dependent variable (total flow) changes with a covariate (total rain) between two groups (stream A & B). Usually, i would go with ANCOVA. But these groups have the ...
2
votes
0answers
12 views

Simple effects analysis - correct univariate results for multiple comparisons?

I tested the effect of two factors with both each three levels on a DV. When analyzing my data, the ANOVA revealed two significant main effects and a significant 3x3 interaction which I now plan to ...
1
vote
1answer
49 views

Is it necessary to do a Bonferroni correction on a) exploratory analysis b) correlations or c) if there are 3 different dependent variables

My study is a 2x2 within design and I have a series of 8 hypotheses. These include one ANOVA , one t-test and 4 Pearson correlations. However, the ANOVA and 3 of the correlations include the same ...
2
votes
1answer
52 views

How Is This Design a Randomized Complete Block Design?

I recently started learning about one-way ANOVA, multiple comparisons, and contrasts. At the moment, I have an introductory-level knowledge of these concepts. I'm analysing an experiment that ...
0
votes
2answers
285 views

Bonferroni correction

I want to validate a questionnaire with three scales summing up to a total score. For for all tests I conducted, I tested the scales separately and additionally the total score (so 3+1 and that total ...
4
votes
1answer
350 views

Difference between pairwise t test and multivariate linear regression results

I got different results when comparing means of different groups using a pairwise t test and multivariate linear regression. ...