# Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1,202 questions
0answers
38 views

### Using test MSE to compare models

I was reading through the definition of MSE and the formula I have found in all articles is the one shown in https://en.wikipedia.org/wiki/Mean_squared_error , which is the expected squared difference ...
1answer
20 views

### Multiple comaprison and secondary research

It's maybe some kind of philosophical question. I have a large dataset with information about customers. I return to the dataset from time to time to make some estimations or check different ...
1answer
58 views

### How to obtain the results of the Nemenyi post-hoc test in a table showing grouped pairs?

I performed a Friedman's test, which showed that there are differences among my treatments. Thus, I choose to perform the Nemenyi's post-hoc test in order to infer which treatments statistically ...
1answer
31 views

### qPCR Gene expression validation and multiple testing

It's common practice to adjust for multiple testing when analyzing high-throughput gene expression data, eg. RNA-seq and microarrays. Despite this, researchers routinely select a subset of genes for ...
1answer
202 views

### emmeans pairwise contrasts result in same output values for all?

First of all, I am a beginner at mixed models, so I beg your patience and advice if this post could be improved. The structure of the data: ...
2answers
46 views

### $R^2$ and p-value for multivariate linear regression with confounders

I am working on a project where I am interested in the following variables: Dependent variables: $y_1,\, y_2,\, y_3,\, y_4,\, y_5$ (continuous) Independent variables: $x_1,\, x_2, \ldots,\, x_{18}$ ...
0answers
17 views

### Indiscpecies - multipatt and overcoming multi-comparrisons

What is the best way to overcome the potential issue of type 1 error when doing an indicator analysis using the multipatt function in the ...
0answers
23 views

### comparison of two mean curves

I have two mean values based on the same sample of people, call them $x_1$ and $x_2$. I know that to compare two means, I would usually use a t-test. However, what I have is a plot of each against a ...
0answers
11 views

### significance test by confidence interval in complete randomization design(CRD)

In page 91 of this book, Design and analysis of experiments by Montgomery , the author stated after Equation (3.30) that: Clearly, if the confidence interval in Equation 3.30 includes zero, ...
1answer
34 views

### How can I test whether an individual treatment mean significant or not?

Suppose by complete randomization design, I reach into a decision that there is at least difference between two treatment means. That is, my hypothesis is $$H_o:\mu_1=\mu_2=\mu_3=\mu_4=\mu_5$$ ...
0answers
26 views

### Is two-factor factorial design most appropriate in this situation?

Suppose there are two factors, A and B. There are 7 levels for factor A and 4 levels of ...
1answer
24 views

### Multiple Comparison Correct Group of Confidence Intervals

I have a number of data points (~10,000) compared against different confidence intervals to determine whether or not they are significantly elevated or not. Now I would like to correct for multiple ...
0answers
13 views

### Comparative Analysis

First I would like to indicate I plan to use Bayesian analysis methods (as instructed) I want to compare the effect of 6 independent variables (IVs) on four dependent variables. The dependent ...
1answer
338 views

### How to interpret the result of Friedman's test?

I uploaded the data and then I used bootstrapping to have 10 different samples from the original data but with the same length as the original data. For each sample, I used 7 distance metrics, and I ...
1answer
40 views

### Type I and Type II errors when varying the number of treatments

I have a hypothetical sample of n observations, and I want to test if there are significant differences across multiple treatments. I need to know how the number of treatments interfere in the type I ...
1answer
26 views

### Post hoc testing without initial ANOVA

I am comparing the response to a drug under five different conditions. The textbook instruction is to perform a one-way ANOVA followed by a post hoc test if necessary. The P value from an ANOVA is ...
1answer
49 views

### How to state my model in R in words (linear mixed effect model, general linear hypothesis, Tukey, Benjamini Hochberg)

I have a dataset that has four treatment groups (AX, AY, BX, BY). I am trying to determine if there is a significant difference in a response variable (Length) between any of these four treatment ...
1answer
23 views

### Which of these scenarios are multiple or repeated comparisons?

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused. My confusion started with A/B testing ...
0answers
25 views

### Which statistical test to determine interaction effect?

I conducted an experiment to determine the effect of blood type on blood cell interactions with proteins of differing blood groups and want to determine if there is an interaction effect for the ...
0answers
8 views

### How to demonstrate that a generation process outperforms some others, given a parameter?

Dear statistical stackexchanger, I would like to evaluate whether an algorithm outperforms others for a task of increasing and increasing complexity. I am measuring the impact of the selected ...
0answers
51 views

### Which 'single-step' multiple comparison test is being employed within R multcomp::glht?

When performing a general linear hypothesis test using the glht function within the R multcomp library. For example: ...
0answers
13 views

### Correcting p-values from jackknife resamples

I have data from few subjects (N=10), compare the DV using ANOVA and get a p-value. Since I have few subjects, I want to see if the significant p-value didn't just happen due to one subject. Ok, I ...
0answers
41 views

### Analogue of critical difference (CD) diagram for the comparison over single dataset

does anybody know any fancy way to present the performance of multiple classifiers (in terms of AUC, for example) over just 1, single dataset? I am familiar with the critical difference diagram for ...
1answer
51 views

### Formulae for FDR values [duplicate]

Can anybody give a formula for the "FDR values"? (I write "FDR values" because strictly speaking, you don't have an FDR value for each test, but an FDR value for the whole study, but nevertheles, you ...
0answers
48 views

### 30 persons, 3 related readouts

I am stuck with how to best analyze the following experimental design: We analyze 3 variables (A, B, C), that are not independent from each other: if one of them is high, the other ones are expected ...
0answers
269 views

### P-value adjustment for hierarchical / multilevel data

I'm dealing with a large data set which is hierarchical in nature, that is 1000 schools, within every school 1000 students. The dependent variable is a measure of performance. Ideally I would like to ...
1answer
19 views

### Comparing same item but before and after result

I have a questionnaire for the effectiveness of the implementation of a short course. Respondents need to answer 6 items in part B of my questionnaire with 4-point likert scale (ordinal data). The ...
1answer
83 views

### 2-way ANOVA multiple comparisons question

My question is about a 2-way ANOVA for an experiment. For simplicity let's say I'm using 3 methods to stimulate cells at 3 different time points, and I want to look at a resulting output variable. ...
0answers
8 views

### Kendalls u-coefficient to calculate overall consistency of pair comparison experiments

I have 8 conditions. In a pairwise experiment, I am comparing 28 pairs: condition1 vs condition2, condition3 vs. condition 4, condition1 vs condition3, condition4 vs, condition2, etc etc. Each ...
1answer
216 views

### Bonferroni correction: how does it work?

I already read the answers here: About the Bonferroni correction, but I still don't understand the concept completely. Assume the following hypothetical case, where I perform an ANOVA test to ...
2answers
41 views

### How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
1answer
73 views

### Pairwise comparisons on mcglm joint model in r

Using the mcglm package in r, I have been able to get pairwise comparisons from single-DV models but have not figured out how to get pairwise comparisons for an IV according to both DVs in a joint ...
1answer
55 views

### Understanding Kenward-Rogers Degrees of Freedom from lsmeans()

I'm trying to analyze data using a multilevel model which predicts subject response times using the experimental group and trial validity. ...
0answers
32 views

### Correct approach to performing multiple comparisons?

Preamble I have conducted an experiment to determine whether my method is sensitive to receiving the true data or random data. In all of my experiments, I worked with a collection of 1,000 data ...
0answers
73 views

### Multivariate hypothesis testing for proportions

It's very common in website optimisation to run A/B tests for conversion rates (e.g. 'buy' button is green by default, would the conversion rate from 'visit' to 'buy' be significantly larger if we ...
0answers
33 views

### What is the difference between seemingly unrelated regression (SUR) and correcting a set of OLS results for multiple comparisons?

As I understand it, the the seemingly unrelated regressions (SUR) or seemingly unrelated regression equations (SURE) models estimate a set of Ordinary Least Squares (OLS) equations where the error ...
0answers
126 views

### Bonferroni correction for pairwise proportion comparisons for more than one outcome level

Suppose I have an event that can have 3 different outcomes. This event is recorded for 21 different factor levels for a total of 100 times, and each time an outcome happens it is counted (so I am ...
0answers
32 views

### Comparing predictiveness of alternative class definitions using linear regression

I am comparing highly-correlated ways to classify patients as sick (1) or well (0). I want to see whether the way you define sickness impacts how well sickness predicts clinical outcomes. I have ...
1answer
72 views

### When testing multiple hypotheses, what does it mean when there are not enough extremes? [closed]

Suppose you are testing a large number of hypotheses, say a million. Unlike the usual situation where you have a lot of very small p-values, in this case all of your p-values are greater than 5%. ...
0answers
44 views

### Compare 2 classifiers on 2 datasets using F1-score

I am doing a little research in the field of multiclass imbalanced data classification. It entails comparing multiple "base" classifiers on 2 datasets. I followed the well known paper by Demsar and ...
0answers
10 views

### Chisquare for homogenity on subsets of samples for more detailed statements?

I have two samples of six treatment groups. The outcome is an amount of positives. The treatments in both samples are the same, the measured outcome is different but dichotomous in both samples. I ...
0answers
19 views

### Should I correct for multiple comparisons after simple main effects?

I had recently a discussion with a colleague and I can't figure out who is right. Suppose you run a study with two factors: A (level 1 and level 2) and B (Level 1, level 2 and level 3). You analyse ...
1answer
48 views

### GLMM summary and multcomp differences

Thanks you for you time to read this question. I run a GLMM because I have measure repeated on 100 individuals. The explanatory variable it is a proporcion so I use binomial family. The final model ...
1answer
69 views

### My deep learning model improved - is the difference significant?

Often deep learning/machine learning models are compared just by measuring their performance on a validation dataset - the model that performs better "wins". Within null hypothesis testing, two ...
0answers
20 views

### Most appropriate statistical test for multiple comparisons with categorical response variable in r

I am looking to compare the call type usage of humpback whales across three different group compositions. There are 13 different call types, and I would like to develop a model that will allow me to ...
0answers
70 views

### Do results from the lme-function require adjustment of p-values?

If I run a linear mixed model with the lme() function and get results like these (comparing the score of 4 treatment groups against a placebo group): ...
0answers
11 views

### Multiple hypothesis testing: continuous vs. discrete treatments

I'm studying the effect of rainfall, cloud cover, and temperature on mood. I also want to examine whether these weather variables interact with each other. These are three separate (continuous) ...
0answers
21 views

### Are p-value corrections needed for separate tests with various grouping variables?

I have a dataset (find an example below) in which there are many factor variables (group_a, group_b, ...
1answer
24 views

### How to test for pairwise differences between factor levels while looking at an interaction with another factor?

Let's say I have a survey with questions 1 through 10 about different characteristics of animals rated from 1 to 10 and this is repeated for 5 different animals. Suppose I collect 50 surveys so my ...
0answers
54 views

### Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...