Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1,191 questions
38 views

How to adjust for the number of nested model tests?

How should I adjust for the number of tests I do when using the likelihood ratio difference test in a model estimated using maximum likelihood (or some flavor of), where the null hypothesis of the ...
22 views

How can I check if three or more results are statistically identical?

I conducted several experiments (for the moment 3, but there will be more in the near future), where in each experiment I made several measurements. I know (from previous experience) that the result ...
62 views

Am I Double-Dipping on my Hypothesis Test?

I am testing a large number of simultaneous hypotheses (~300), with each hypothesis test trying to determine whether a given test statistic differs significantly from zero. Am I cheating the multiple ...
53 views

Is scalar invariance required to compare relationships between groups (but not comparing differences in means)

I am undertaking multigroup comparisons with the aim of comparing structural relationships between groups. I have established configural and metric invariance but do not have partial scalar invariance....
31 views

Help with power analyses - adjusting alpha level correctly

I will be conducting a series of ANCOVAs and MANCOVAs for a 2 x 3 x 3 design with 2 covariates (1 covariate will only be relevant for a subset of analyses). There are 9 main DVs in the study. Three of ...
112 views

Drawing conclusions of several inferences with the same data in one study

In a sample size of 100, we identified the existence of two attributes A and B. Our goal is to assess whether there is any ...
56 views

How to calculate % information in longitudinal group sequential design

I'm looking at using group sequential design for a study that will treat subjects with drug or placebo and then follow them for a year. The subjects get measured every 3 months for the year under ...
85 views

Wilcoxon Signed Rank Test for equally likely groups

I have a data set with results from 50 subjects who each tested two different types of running shoe, A and B, on four different surfaces, (i.e. each subject conducted 8 tests each). After each test ...
124 views

Adjust p-values returned from a binomial test for multiple testing

I'm using the binomial test in R to test whether the number of observations of X are significantly higher or lower than expected. e.g.: ...
30 views

Correction for multiple comparisons after permutation

I've trained 17 regression models using a support vector regression machine providing correlations (r) between true and predicted labels and the mean squared error (MSE) for each model. The target ...
304 views

About Kruskal-Wallis, Mann-Whitney U and multiple comparisons correction

I have 5 independent samples which I want to compare, to define if there are significant differences in the distribution of values across samples. The samples' sizes are: 4562, 1116, 314, 151, 77 I ...
45 views

How to visualize and represent many (>20) post-hoc hypothesis tests and p-values?

I have a repeated measure design with 2 factors, one of which has 3 levels and the other has 4 levels. This results in 66 pairwise comparisons when I use lsmeans()...
25 views

Multiple comparisons for multiple methods on multiple independent datasets

I have some problems in mutliple comparisons. I am building a regression-based model to predict the scores of some incoming texts. In one of my experiments, I propose the hypothesis that modelling ...
31 views

mixed / hierarchical /multilevel modelling: questions about model design

I'm new to multilevel modelling, so these are (hopefully) basic questions. :) I'll first describe the dataset and a proposed model, followed by questions of the model. The data: I have a S subjects ...
117 views

two-way ANOVA - how to conduct post-hoc tests?

Conducted a two-way ANOVA and got a main effect of sound manipulation onto perceived diagnosticity. ...
39 views

Compare means between months of consecutive years

I have sales data of 150 sellers of an Internet website. For each seller I have daily sales for two years, that is, 730 measures, and there are quite a few days in which the sales are 0. On the other ...
105 views

Controlling FDR when p-values are bimodal

I have a quick question about using the Benjamini-Hochberg procedure to control FDR using the p.adjust() function in R. I have read that in order to properly apply ...
69 views

How and when to use multiple comparison tests

There seems to be some variability in WHEN multiple comparison tests such as Bonferroni or FDR might be implemented. Type 1: One might carry out multiple one-way ANOVA's to see whether or not there ...
30 views

P-value corrections when distribution is bimodal

I am analyzing a set of transcriptome data to assess which "GO categories" are significantly enriched using a tool called GOSeq (https://bioconductor.org/packages/release/bioc/html/goseq.html). Put ...
45 views

How to compare several means to control when the distribution is not normal and sample sizes are unequal?

I have a large dataset (n = 170,000), distributed roughly equally among 5 groups (each is 34,000 ± ~1,000). I want to compare whether the means in any of the first 4 groups are different from the ...
59 views

FDR and the the Benjamini-Hochberg Method

I am trying to understand the Benjamini-Hochberg Method for controlling the false discovery rate. Mathematically, if we are given with m hypothesis testing procedures, we sort the P-Values and reject ...
1k views

Why do planned comparisons and post-hoc tests differ?

I conducted a three-way independent ANOVA. I manipulated the sound of my stimulus to be 1: congruent (KON), 2: incongruent (INK) and 3: no sound at all (control condition). One of my planned ...
19 views

How to compare three groups of data each with a different sample size

I have a sample of 26 individuals. Those were requested to answer a questionnaire. Based on the collected info I have arranged my sample in 3 different groups: one with 16 individuals - GROUP A, ...
44 views

Monte-Carlo Simulation Between Different Variables in the Same Model

I'd like to run a Monte Carlo simulation to test the null hypothesis that the predicted probabilities of two binary independent variables which are a part of my logistic regression model are equal ...
14 views

How to test hypothesis with multiple 'or' restictions

I am seeking help to interpret a question and set up test procedures for a series of hypotheses (six in total, two of which I can answer). It is part of a weekly problem set we are given in class. The ...
23 views

Do I need to correct for multiple testing if I perform one mediation model with 3 parallel mediators?

I am performing a mediation model with 3 parallel mediators, 1 independent variable and 1 dependent variable. My question is: is this considered a unique analysis, or should I control for multiple ...
57 views

why correct for multiple comparisons in a correlation table?

I often read that in a correlation matrix the p-values for all correlations in a NxN matrix should be adjusted for multiple comparisons (for example: Multiple comparisons for correlation matrix?). ...
13 views

In multiple restriction alternative hypothesis, should this be AND/OR or just OR?

Consider a null hypothesis under two restrictions: $H_0: \beta_1=u, \beta_2=v$ I encounter two ways to form the alternative hypothesis. One is: $H_A: \beta_1 \ne u$ $OR$ $\beta_2 \ne v (1)$ The ...
93 views

A few questions regarding the practice of heterogeneous treatment effect analysis (a.k.a, interaction detection or subgroup analysis) methods

Imagine I am looking at a randomized experiment between a control and one or more treatment conditions. For example, I have a treatment that aims to get people out of debt. I randomize people to ...
53 views

Using local FDR and q-values in practice?

I've learned that the local FDR and q-values are the estimated FDR given a fixed rejection region, which is the opposite of other frequentist methods where you fix the error rate first to estimate the ...
23 views

Bayesian statistic in model comparison

I am trying to use the Bayesian statistic in model comparison. Let's say I have 4 different pharmacokinetic models which I use to fit breast MRI data. Each of this model M gives me a certain number of ...
84 views

Multiple Test Correction: Can we divide dataset into subgroups?

Assume that the researcher wants to test huge amount of hypothesis and present his discoveries. He has data from two groups, eg, 100 people vs 100 people, and he wants to 1) having 20.000 genes ...
42 views

Is it correct to do pre-filtering before multiple tests?

I often see that authors of large-scale studies that theoretically have to perform multiple test correction for enormous amount of tests do pre-filtering, in other words - throw away all tests that ...
13 views

Statistical test to compare cellular types

I have a statistical question. I have 10 patients and 5 cellular lines. For each patient the gene expression of 6 genes was measured. Imagine the y-axis represents the expression of that genes in ...
20 views

Using q-values in practice?

I have been trying to wrap my head around the concept of q-values for a while, but I'm fairly new to statistics so have been struggling with understanding papers. From Storey's paper, I'm given to an ...
104 views

Comparing different treatments using logistic regression

I have some data on testing different pesticides on insects. I have applied different concentrations and different pesticides (total 11 treatments) for 24 hours, and have 6 different data points. I'm ...
65 views

Adjust for multiple comparisons in several mixed models

I am fitting several generalized mixed models with binomial family to model proportions data(dependent variable) and 3 factors.The data looks like this: Sample Day ligand Condition Conc1 Conc2 .... ...
110 views

52 views

How Is This Design a Randomized Complete Block Design?

I recently started learning about one-way ANOVA, multiple comparisons, and contrasts. At the moment, I have an introductory-level knowledge of these concepts. I'm analysing an experiment that ...
21 views

How to match more than two samples?

I'd like to compare the average values of some variables between 3 different groups. I should control for some covariates between the members of these 3 groups. For that, I decided to use matching ...
23 views

Advice for conducting the same experiment several times?

I'm very new to Bayesian statistics, but I've found myself in a situation where they might be able to help. I'll be running the same experiment repeatedly and analyzing the data every few months. The ...
199 views

Interpreting odds ratio of multiple comparisons from a logistic regression model (using R)

I recently ran a logistic regression on categorical data and ran a Tukey multiple comparisons post hoc analysis using the glht ...
16 views

How can I create and analyze a composite score for multiple pre- and post-intervention variables?

I have a relatively small sample (~30) for an exercise intervention study. I have pre and post intervention data for three variables that are all related to cardiovascular health but are not on the ...
25 views

How to compare curves behavior?

I want to know how I can compare the "behavior of curves" and see what curves behave more similar. In the link below, in the 1st graph we have 4 curves through out time. http://www.sooty.ca/sooty/BDI....
132 views

Adjustment for multiple tests for a mediation with multiple DVs? (Process/SPSS)

I hope my question has not been addressed in another post; I could not find any response to my issue. I'm conducting a mediation analysis with 1 IV (X), 1 mediator (M), and 4 DVs (Y). I use Process, ...
5 views

how to determine the number of comparisons in multiple testing p value adjustment

I have a dataset of 100 biomarkers. For each biomarker, there are 3 groups. In my analysis, I did pairwise ttest between each group and generate p-values for each biomarker. And I did this for all ...