Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
38 views

Using test MSE to compare models

I was reading through the definition of MSE and the formula I have found in all articles is the one shown in https://en.wikipedia.org/wiki/Mean_squared_error , which is the expected squared difference ...
0
votes
1answer
20 views

Multiple comaprison and secondary research

It's maybe some kind of philosophical question. I have a large dataset with information about customers. I return to the dataset from time to time to make some estimations or check different ...
0
votes
1answer
58 views

How to obtain the results of the Nemenyi post-hoc test in a table showing grouped pairs?

I performed a Friedman's test, which showed that there are differences among my treatments. Thus, I choose to perform the Nemenyi's post-hoc test in order to infer which treatments statistically ...
0
votes
1answer
31 views

qPCR Gene expression validation and multiple testing

It's common practice to adjust for multiple testing when analyzing high-throughput gene expression data, eg. RNA-seq and microarrays. Despite this, researchers routinely select a subset of genes for ...
1
vote
1answer
202 views

emmeans pairwise contrasts result in same output values for all?

First of all, I am a beginner at mixed models, so I beg your patience and advice if this post could be improved. The structure of the data: ...
1
vote
2answers
46 views

$R^2$ and p-value for multivariate linear regression with confounders

I am working on a project where I am interested in the following variables: Dependent variables: $y_1,\, y_2,\, y_3,\, y_4,\, y_5$ (continuous) Independent variables: $x_1,\, x_2, \ldots,\, x_{18}$ ...
1
vote
0answers
17 views

Indiscpecies - multipatt and overcoming multi-comparrisons

What is the best way to overcome the potential issue of type 1 error when doing an indicator analysis using the multipatt function in the ...
1
vote
0answers
23 views

comparison of two mean curves

I have two mean values based on the same sample of people, call them $x_1$ and $x_2$. I know that to compare two means, I would usually use a t-test. However, what I have is a plot of each against a ...
0
votes
0answers
11 views

significance test by confidence interval in complete randomization design(CRD)

In page 91 of this book, Design and analysis of experiments by Montgomery , the author stated after Equation (3.30) that: Clearly, if the confidence interval in Equation 3.30 includes zero, ...
0
votes
1answer
34 views

How can I test whether an individual treatment mean significant or not?

Suppose by complete randomization design, I reach into a decision that there is at least difference between two treatment means. That is, my hypothesis is $$H_o:\mu_1=\mu_2=\mu_3=\mu_4=\mu_5$$ $$...
0
votes
0answers
26 views

Is two-factor factorial design most appropriate in this situation?

Suppose there are two factors, A and B. There are 7 levels for factor A and 4 levels of ...
1
vote
1answer
24 views

Multiple Comparison Correct Group of Confidence Intervals

I have a number of data points (~10,000) compared against different confidence intervals to determine whether or not they are significantly elevated or not. Now I would like to correct for multiple ...
0
votes
0answers
13 views

Comparative Analysis

First I would like to indicate I plan to use Bayesian analysis methods (as instructed) I want to compare the effect of 6 independent variables (IVs) on four dependent variables. The dependent ...
3
votes
1answer
338 views

How to interpret the result of Friedman's test?

I uploaded the data and then I used bootstrapping to have 10 different samples from the original data but with the same length as the original data. For each sample, I used 7 distance metrics, and I ...
2
votes
1answer
40 views

Type I and Type II errors when varying the number of treatments

I have a hypothetical sample of n observations, and I want to test if there are significant differences across multiple treatments. I need to know how the number of treatments interfere in the type I ...
0
votes
1answer
26 views

Post hoc testing without initial ANOVA

I am comparing the response to a drug under five different conditions. The textbook instruction is to perform a one-way ANOVA followed by a post hoc test if necessary. The P value from an ANOVA is ...
0
votes
1answer
49 views

How to state my model in R in words (linear mixed effect model, general linear hypothesis, Tukey, Benjamini Hochberg)

I have a dataset that has four treatment groups (AX, AY, BX, BY). I am trying to determine if there is a significant difference in a response variable (Length) between any of these four treatment ...
2
votes
1answer
23 views

Which of these scenarios are multiple or repeated comparisons?

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused. My confusion started with A/B testing ...
1
vote
0answers
25 views

Which statistical test to determine interaction effect?

I conducted an experiment to determine the effect of blood type on blood cell interactions with proteins of differing blood groups and want to determine if there is an interaction effect for the ...
1
vote
0answers
8 views

How to demonstrate that a generation process outperforms some others, given a parameter?

Dear statistical stackexchanger, I would like to evaluate whether an algorithm outperforms others for a task of increasing and increasing complexity. I am measuring the impact of the selected ...
0
votes
0answers
51 views

Which 'single-step' multiple comparison test is being employed within R multcomp::glht?

When performing a general linear hypothesis test using the glht function within the R multcomp library. For example: ...
0
votes
0answers
13 views

Correcting p-values from jackknife resamples

I have data from few subjects (N=10), compare the DV using ANOVA and get a p-value. Since I have few subjects, I want to see if the significant p-value didn't just happen due to one subject. Ok, I ...
0
votes
0answers
41 views

Analogue of critical difference (CD) diagram for the comparison over single dataset

does anybody know any fancy way to present the performance of multiple classifiers (in terms of AUC, for example) over just 1, single dataset? I am familiar with the critical difference diagram for ...
1
vote
1answer
51 views

Formulae for FDR values [duplicate]

Can anybody give a formula for the "FDR values"? (I write "FDR values" because strictly speaking, you don't have an FDR value for each test, but an FDR value for the whole study, but nevertheles, you ...
0
votes
0answers
48 views

30 persons, 3 related readouts

I am stuck with how to best analyze the following experimental design: We analyze 3 variables (A, B, C), that are not independent from each other: if one of them is high, the other ones are expected ...
0
votes
0answers
269 views

P-value adjustment for hierarchical / multilevel data

I'm dealing with a large data set which is hierarchical in nature, that is 1000 schools, within every school 1000 students. The dependent variable is a measure of performance. Ideally I would like to ...
1
vote
1answer
19 views

Comparing same item but before and after result

I have a questionnaire for the effectiveness of the implementation of a short course. Respondents need to answer 6 items in part B of my questionnaire with 4-point likert scale (ordinal data). The ...
1
vote
1answer
83 views

2-way ANOVA multiple comparisons question

My question is about a 2-way ANOVA for an experiment. For simplicity let's say I'm using 3 methods to stimulate cells at 3 different time points, and I want to look at a resulting output variable. ...
1
vote
0answers
8 views

Kendalls u-coefficient to calculate overall consistency of pair comparison experiments

I have 8 conditions. In a pairwise experiment, I am comparing 28 pairs: condition1 vs condition2, condition3 vs. condition 4, condition1 vs condition3, condition4 vs, condition2, etc etc. Each ...
2
votes
1answer
216 views

Bonferroni correction: how does it work?

I already read the answers here: About the Bonferroni correction, but I still don't understand the concept completely. Assume the following hypothetical case, where I perform an ANOVA test to ...
0
votes
2answers
41 views

How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
0
votes
1answer
73 views

Pairwise comparisons on mcglm joint model in r

Using the mcglm package in r, I have been able to get pairwise comparisons from single-DV models but have not figured out how to get pairwise comparisons for an IV according to both DVs in a joint ...
1
vote
1answer
55 views

Understanding Kenward-Rogers Degrees of Freedom from lsmeans()

I'm trying to analyze data using a multilevel model which predicts subject response times using the experimental group and trial validity. ...
0
votes
0answers
32 views

Correct approach to performing multiple comparisons?

Preamble I have conducted an experiment to determine whether my method is sensitive to receiving the true data or random data. In all of my experiments, I worked with a collection of 1,000 data ...
0
votes
0answers
73 views

Multivariate hypothesis testing for proportions

It's very common in website optimisation to run A/B tests for conversion rates (e.g. 'buy' button is green by default, would the conversion rate from 'visit' to 'buy' be significantly larger if we ...
1
vote
0answers
33 views

What is the difference between seemingly unrelated regression (SUR) and correcting a set of OLS results for multiple comparisons?

As I understand it, the the seemingly unrelated regressions (SUR) or seemingly unrelated regression equations (SURE) models estimate a set of Ordinary Least Squares (OLS) equations where the error ...
0
votes
0answers
126 views

Bonferroni correction for pairwise proportion comparisons for more than one outcome level

Suppose I have an event that can have 3 different outcomes. This event is recorded for 21 different factor levels for a total of 100 times, and each time an outcome happens it is counted (so I am ...
1
vote
0answers
32 views

Comparing predictiveness of alternative class definitions using linear regression

I am comparing highly-correlated ways to classify patients as sick (1) or well (0). I want to see whether the way you define sickness impacts how well sickness predicts clinical outcomes. I have ...
4
votes
1answer
72 views

When testing multiple hypotheses, what does it mean when there are not enough extremes? [closed]

Suppose you are testing a large number of hypotheses, say a million. Unlike the usual situation where you have a lot of very small p-values, in this case all of your p-values are greater than 5%. ...
0
votes
0answers
44 views

Compare 2 classifiers on 2 datasets using F1-score

I am doing a little research in the field of multiclass imbalanced data classification. It entails comparing multiple "base" classifiers on 2 datasets. I followed the well known paper by Demsar and ...
1
vote
0answers
10 views

Chisquare for homogenity on subsets of samples for more detailed statements?

I have two samples of six treatment groups. The outcome is an amount of positives. The treatments in both samples are the same, the measured outcome is different but dichotomous in both samples. I ...
0
votes
0answers
19 views

Should I correct for multiple comparisons after simple main effects?

I had recently a discussion with a colleague and I can't figure out who is right. Suppose you run a study with two factors: A (level 1 and level 2) and B (Level 1, level 2 and level 3). You analyse ...
1
vote
1answer
48 views

GLMM summary and multcomp differences

Thanks you for you time to read this question. I run a GLMM because I have measure repeated on 100 individuals. The explanatory variable it is a proporcion so I use binomial family. The final model ...
2
votes
1answer
69 views

My deep learning model improved - is the difference significant?

Often deep learning/machine learning models are compared just by measuring their performance on a validation dataset - the model that performs better "wins". Within null hypothesis testing, two ...
0
votes
0answers
20 views

Most appropriate statistical test for multiple comparisons with categorical response variable in r

I am looking to compare the call type usage of humpback whales across three different group compositions. There are 13 different call types, and I would like to develop a model that will allow me to ...
4
votes
0answers
70 views

Do results from the lme-function require adjustment of p-values?

If I run a linear mixed model with the lme() function and get results like these (comparing the score of 4 treatment groups against a placebo group): ...
0
votes
0answers
11 views

Multiple hypothesis testing: continuous vs. discrete treatments

I'm studying the effect of rainfall, cloud cover, and temperature on mood. I also want to examine whether these weather variables interact with each other. These are three separate (continuous) ...
2
votes
0answers
21 views

Are p-value corrections needed for separate tests with various grouping variables?

I have a dataset (find an example below) in which there are many factor variables (group_a, group_b, ...
1
vote
1answer
24 views

How to test for pairwise differences between factor levels while looking at an interaction with another factor?

Let's say I have a survey with questions 1 through 10 about different characteristics of animals rated from 1 to 10 and this is repeated for 5 different animals. Suppose I collect 50 surveys so my ...
3
votes
0answers
54 views

Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...