Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
8 views

How to compensate for different statistical power in multiple comparisons?

Imagine we are performing 10 comparisons on a single variable of choice. The 10 respective experiments have different statistical powers. Now imagine that the least powerful and the most powerful ...
0
votes
1answer
33 views

Generalization of one-sample t-test for multiple groups

If I have some data*, and want to test the hypothesis that it has a given mean $\mu_0$, I know of the possibility of using a one-sample t-test. Here, $\mu_0=0$ could be the null-hypothesis of a ...
0
votes
1answer
9 views

Appropriate Multiple Testing Correction for Correlation Matrix

I have $N$ samples of $D$ variables, for which I want to study the correlation. Specifically, I want to know if any of these variables are correlated. I can get estimates and a matrix of p-values in R ...
1
vote
0answers
16 views

Pairwise non-parametric test?

I have 5 groups of patients (different sizes), for which I measured some property that is continuous but not normally distributed. I would like to compare every pair of groups in respect to this ...
0
votes
1answer
25 views

Correcting for multiple post hoc tests in factorial ANOVA

I have done a two-way repeated-measures ANOVA with factor A having 3 levels and factor B having 5 levels. I am doing post hoc comparisons between all 3 levels of factor A within each of the 5 levels ...
1
vote
4answers
49 views

Why not remove an experimental group and do a t-Test instead of an ANOVA?

This has never been made clear to me before, so I would love some help. Lets say I have 3 experimental groups of animals (A,B,C) A is the baseline control, B is treatment x, and C is treatment y. I ...
1
vote
0answers
26 views

Pairwise comparisons after ANOVA with Tukey and Bonferroni methods in r

The built-in functions TukeyHSD and LSD.test from R perform pairwise comparisons. In order to maintain the type I error, the significance level $0.05$ is divided by the number of comparisons $k$. Do ...
0
votes
0answers
19 views

Tukey test for mixed model not giving all the comparisons [closed]

I am comparing the number of lesions in four different quadrants (each facing one cardinal point in 15 trees. My variables are: hits= number of lesions. continuous variable. response variavle ...
1
vote
0answers
13 views

FIsher's LSD test is appropriate for restricted mean comparisons

Suppose that we perform two way ANOVA with interactions, with factors $A$ and $B$ with three level each $A_{1},A_{2},A_{3}$ and $B_{1},B_{2},B_{3}$. We know that Fisher's LSD test is used for ...
2
votes
1answer
25 views

Repeated measures models multiple testing adjustment

Perhaps I am missing something obvious here, but I can't find an answer to this. Let's say we run 10 individual repeated measures models with factor(time) as fixed effect with 5 separate timepoints ...
0
votes
0answers
10 views

Are differences in proportions meaningful in a sample's segmentation

I am setting up an A/B test of a webpage where users will be randomly distributed between either A or B. I'll be using the N-1 Proportion Test to determine which page converts better than the other. ...
0
votes
1answer
24 views

Unsure which data to use for ANOVA test

I have determined that a one-way ANOVA test of variances will work best for my data. However, the nature of my data makes me very concerned about making any claims based on significance or not. My ...
1
vote
0answers
29 views

Model selection using p-values - tree inference

Suppose I have some i.i.d. normal observations from $\mathbb{R}^f$ with parameters $(\mu, \Sigma)$ and $\Sigma$ is known to be the identity matrix. I have the following hypotheses: $H_0^i$: $\mu_i = ...
0
votes
0answers
9 views

Most suitable comparison between groups

I have a neural network based algorithm which it was trained with 5 different datasets (A, B, C, D and E). For each one, 10 training runs was made {(A1, A2, ..., A10), (B1, ..., B10), ...}. Then, it ...
0
votes
1answer
24 views

Best way to compare several cases to a null model

At the moment I am doing analysis in some data that is (currently) divided by number of individuals in a group. More or less like: Groups w/ two individuals: Value_1, Value_2, ..., Value_N Groups w/ ...
4
votes
3answers
99 views

Fairness of 6 sided die NOT using Chi Square Test

So I was reading a lot of online articles using the chi-square test to determine the fairness of a die. Correct me if I am wrong, but isn't the drawbacks of using a goodness of fit test is that you ...
0
votes
0answers
13 views

Sample size calculation for multiple hypotheses testing on two or more metrics

Let's say I want to run an experiment where there's a control and there's a feature group (say, 50-50 random split). The treatment can affect the feature group in 1, 2 or three ways (or any ...
0
votes
1answer
31 views

Post-hoc corrections for multiple t-tests

I am comparing two groups (young vs. old adults) for differences in 5 brain regions (size) with independent t-tests. I also compare performance on three different aspects of a task with independent t-...
0
votes
0answers
12 views

Correct group of factorial and repeated-measure ANOVA

I'm currently having trouble defining what type of ANOVA I need to run as well as how to run it correctly. This is the following setup: I have 4 groups of 30 people doing a 2-session task (one after ...
0
votes
0answers
23 views

Pairwise comparisons for a nested effects

This is basically a theoretical question which is based on the following R output. I have the following R output which contains both crossed and nested effects. The group(3 levels) and phase(2 levels)...
0
votes
0answers
16 views

Any AI/ML approaches for analysis of competing hypotheses?

I know there are significant development about sentiment analysis in text e.g. "x is awful", "x is bad" or "y is fantastic". But are there similar approaches to compare hypotheses? e.g. given a ...
1
vote
1answer
55 views

Correcting for multiple pairwise comparisons with GAM objects {mgcv} in R

My question concerns pairwise comparisons of factor levels in a gam object. I have a dataframe, df, containing reaction time data (in ms) to stimuli varying in ...
0
votes
0answers
10 views

Combining p-values for parwise tests (single cell data)

I'm working on single-cell data. To find genes which are markers of groups, I perform pairwise comparison between groups of cells. I was used to take the maximal FDR as a criterium to decide if a ...
0
votes
0answers
9 views

Permutation and multiple testing - permuting the test labels instead of sample labels?

I'm comparing $m$ p-values, where $m$ is very large. Every p-value comes from a two-sample $t$-test of two groups of size $n_1$ and $n_2$, where the $n_i$ are very small ($2$ or $3$). This leads to a $...
0
votes
0answers
15 views

Problem of multiple testing when checking the assumptions of a particular test

I have two sets of measures and I'd like to check whether the two samples come from the populations with the equal means. In order to choose the right test, I should first check a few things, such as: ...
0
votes
0answers
15 views

How to calculate false discovery rate for this situation

Let's say I have run 20 mediation models using structural equation modeling and have found evidence for significant effects of interest (i.e., mediational/indirect paths) in 5 out of 20 models. These ...
2
votes
0answers
34 views

Which test to use to detect if a process has significantly more errors than another?

A have a set of machines running a particular software. I want to do something that's called "canary deployment". This consists of deploying the new version of the software to a subset of machines ...
1
vote
1answer
41 views

Adjusting p-values for multiple ANOVAs

I am performing multiple (around 40) multi-way ANOVAs on a dataset, one for each outcome variable. There are three factors involved: cohort, pre/post, and tissue type. For each outcome variable, I ...
0
votes
0answers
18 views

Do independent t-tests on multiple dependent variables for the same subjects need correction for Type I error?

My experiment has one independent variable with 2 levels (sham & surgery) and I have run a PCR experiment on several genes for the subjects in each group. So, each gene is its own dependent ...
1
vote
1answer
60 views

Bonferroni-type corrections and Fisher Method Meta-analysis

I'm not an expert on the field, but I need to know more about how to combined multiple significance test p-values after a two sample Kolmogorov Smirnov (KS)test on two distributions. But I have two ...
0
votes
0answers
35 views

Using test MSE to compare models

I was reading through the definition of MSE and the formula I have found in all articles is the one shown in https://en.wikipedia.org/wiki/Mean_squared_error , which is the expected squared difference ...
0
votes
1answer
19 views

Multiple comaprison and secondary research

It's maybe some kind of philosophical question. I have a large dataset with information about customers. I return to the dataset from time to time to make some estimations or check different ...
0
votes
1answer
43 views

How to obtain the results of the Nemenyi post-hoc test in a table showing grouped pairs?

I performed a Friedman's test, which showed that there are differences among my treatments. Thus, I choose to perform the Nemenyi's post-hoc test in order to infer which treatments statistically ...
0
votes
1answer
21 views

qPCR Gene expression validation and multiple testing

It's common practice to adjust for multiple testing when analyzing high-throughput gene expression data, eg. RNA-seq and microarrays. Despite this, researchers routinely select a subset of genes for ...
1
vote
1answer
111 views

emmeans pairwise contrasts result in same output values for all?

First of all, I am a beginner at mixed models, so I beg your patience and advice if this post could be improved. The structure of the data: ...
1
vote
2answers
43 views

$R^2$ and p-value for multivariate linear regression with confounders

I am working on a project where I am interested in the following variables: Dependent variables: $y_1,\, y_2,\, y_3,\, y_4,\, y_5$ (continuous) Independent variables: $x_1,\, x_2, \ldots,\, x_{18}$ ...
0
votes
0answers
19 views

How to compare 4 different time series and check for statistical significance among them?

I have a data comprising the proportion of a certain bat behavior across nights from 1800h to 1930h. I want to compare and check whether there is a difference in the behavior with 4 varying moon ...
1
vote
0answers
13 views

Indiscpecies - multipatt and overcoming multi-comparrisons

What is the best way to overcome the potential issue of type 1 error when doing an indicator analysis using the multipatt function in the ...
1
vote
0answers
23 views

comparison of two mean curves

I have two mean values based on the same sample of people, call them $x_1$ and $x_2$. I know that to compare two means, I would usually use a t-test. However, what I have is a plot of each against a ...
0
votes
0answers
11 views

significance test by confidence interval in complete randomization design(CRD)

In page 91 of this book, Design and analysis of experiments by Montgomery , the author stated after Equation (3.30) that: Clearly, if the confidence interval in Equation 3.30 includes zero, ...
0
votes
1answer
33 views

How can I test whether an individual treatment mean significant or not?

Suppose by complete randomization design, I reach into a decision that there is at least difference between two treatment means. That is, my hypothesis is $$H_o:\mu_1=\mu_2=\mu_3=\mu_4=\mu_5$$ $$...
0
votes
0answers
19 views

Is two-factor factorial design most appropriate in this situation?

Suppose there are two factors, A and B. There are 7 levels for factor A and 4 levels of ...
1
vote
1answer
24 views

Multiple Comparison Correct Group of Confidence Intervals

I have a number of data points (~10,000) compared against different confidence intervals to determine whether or not they are significantly elevated or not. Now I would like to correct for multiple ...
0
votes
0answers
13 views

Comparative Analysis

First I would like to indicate I plan to use Bayesian analysis methods (as instructed) I want to compare the effect of 6 independent variables (IVs) on four dependent variables. The dependent ...
0
votes
0answers
13 views

Multiple Comparison in non-parametric setting?

I have the following problem: There are 4 independent non-Gaussian continuous distributions, A, B, C, D, and I want to see whether one of A and B is "superior" to both of C and D. I'm aware of large ...
3
votes
1answer
263 views

How to interpret the result of Friedman's test?

I uploaded the data and then I used bootstrapping to have 10 different samples from the original data but with the same length as the original data. For each sample, I used 7 distance metrics, and I ...
2
votes
1answer
34 views

Type I and Type II errors when varying the number of treatments

I have a hypothetical sample of n observations, and I want to test if there are significant differences across multiple treatments. I need to know how the number of treatments interfere in the type I ...
0
votes
1answer
25 views

Post hoc testing without initial ANOVA

I am comparing the response to a drug under five different conditions. The textbook instruction is to perform a one-way ANOVA followed by a post hoc test if necessary. The P value from an ANOVA is ...
0
votes
1answer
48 views

How to state my model in R in words (linear mixed effect model, general linear hypothesis, Tukey, Benjamini Hochberg)

I have a dataset that has four treatment groups (AX, AY, BX, BY). I am trying to determine if there is a significant difference in a response variable (Length) between any of these four treatment ...
2
votes
1answer
22 views

Which of these scenarios are multiple or repeated comparisons?

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused. My confusion started with A/B testing ...