Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

1
vote
0answers
38 views

How to adjust for the number of nested model tests?

How should I adjust for the number of tests I do when using the likelihood ratio difference test in a model estimated using maximum likelihood (or some flavor of), where the null hypothesis of the ...
-1
votes
1answer
22 views

How can I check if three or more results are statistically identical?

I conducted several experiments (for the moment 3, but there will be more in the near future), where in each experiment I made several measurements. I know (from previous experience) that the result ...
3
votes
1answer
62 views

Am I Double-Dipping on my Hypothesis Test?

I am testing a large number of simultaneous hypotheses (~300), with each hypothesis test trying to determine whether a given test statistic differs significantly from zero. Am I cheating the multiple ...
0
votes
0answers
53 views

Is scalar invariance required to compare relationships between groups (but not comparing differences in means)

I am undertaking multigroup comparisons with the aim of comparing structural relationships between groups. I have established configural and metric invariance but do not have partial scalar invariance....
0
votes
0answers
31 views

Help with power analyses - adjusting alpha level correctly

I will be conducting a series of ANCOVAs and MANCOVAs for a 2 x 3 x 3 design with 2 covariates (1 covariate will only be relevant for a subset of analyses). There are 9 main DVs in the study. Three of ...
6
votes
2answers
112 views

Drawing conclusions of several inferences with the same data in one study

In a sample size of 100, we identified the existence of two attributes A and B. Our goal is to assess whether there is any ...
1
vote
1answer
56 views

How to calculate % information in longitudinal group sequential design

I'm looking at using group sequential design for a study that will treat subjects with drug or placebo and then follow them for a year. The subjects get measured every 3 months for the year under ...
1
vote
0answers
85 views

Wilcoxon Signed Rank Test for equally likely groups

I have a data set with results from 50 subjects who each tested two different types of running shoe, A and B, on four different surfaces, (i.e. each subject conducted 8 tests each). After each test ...
0
votes
0answers
124 views

Adjust p-values returned from a binomial test for multiple testing

I'm using the binomial test in R to test whether the number of observations of X are significantly higher or lower than expected. e.g.: ...
1
vote
0answers
30 views

Correction for multiple comparisons after permutation

I've trained 17 regression models using a support vector regression machine providing correlations (r) between true and predicted labels and the mean squared error (MSE) for each model. The target ...
1
vote
1answer
304 views

About Kruskal-Wallis, Mann-Whitney U and multiple comparisons correction

I have 5 independent samples which I want to compare, to define if there are significant differences in the distribution of values across samples. The samples' sizes are: 4562, 1116, 314, 151, 77 I ...
0
votes
0answers
45 views

How to visualize and represent many (>20) post-hoc hypothesis tests and p-values?

I have a repeated measure design with 2 factors, one of which has 3 levels and the other has 4 levels. This results in 66 pairwise comparisons when I use lsmeans()...
0
votes
0answers
25 views

Multiple comparisons for multiple methods on multiple independent datasets

I have some problems in mutliple comparisons. I am building a regression-based model to predict the scores of some incoming texts. In one of my experiments, I propose the hypothesis that modelling ...
0
votes
0answers
31 views

mixed / hierarchical /multilevel modelling: questions about model design

I'm new to multilevel modelling, so these are (hopefully) basic questions. :) I'll first describe the dataset and a proposed model, followed by questions of the model. The data: I have a S subjects ...
1
vote
1answer
117 views

two-way ANOVA - how to conduct post-hoc tests?

Conducted a two-way ANOVA and got a main effect of sound manipulation onto perceived diagnosticity. ...
2
votes
0answers
39 views

Compare means between months of consecutive years

I have sales data of 150 sellers of an Internet website. For each seller I have daily sales for two years, that is, 730 measures, and there are quite a few days in which the sales are 0. On the other ...
1
vote
1answer
105 views

Controlling FDR when p-values are bimodal

I have a quick question about using the Benjamini-Hochberg procedure to control FDR using the p.adjust() function in R. I have read that in order to properly apply ...
1
vote
0answers
69 views

How and when to use multiple comparison tests

There seems to be some variability in WHEN multiple comparison tests such as Bonferroni or FDR might be implemented. Type 1: One might carry out multiple one-way ANOVA's to see whether or not there ...
0
votes
0answers
30 views

P-value corrections when distribution is bimodal

I am analyzing a set of transcriptome data to assess which "GO categories" are significantly enriched using a tool called GOSeq (https://bioconductor.org/packages/release/bioc/html/goseq.html). Put ...
0
votes
1answer
45 views

How to compare several means to control when the distribution is not normal and sample sizes are unequal?

I have a large dataset (n = 170,000), distributed roughly equally among 5 groups (each is 34,000 ± ~1,000). I want to compare whether the means in any of the first 4 groups are different from the ...
2
votes
0answers
59 views

FDR and the the Benjamini-Hochberg Method

I am trying to understand the Benjamini-Hochberg Method for controlling the false discovery rate. Mathematically, if we are given with m hypothesis testing procedures, we sort the P-Values and reject ...
2
votes
1answer
1k views

Why do planned comparisons and post-hoc tests differ?

I conducted a three-way independent ANOVA. I manipulated the sound of my stimulus to be 1: congruent (KON), 2: incongruent (INK) and 3: no sound at all (control condition). One of my planned ...
0
votes
0answers
19 views

How to compare three groups of data each with a different sample size

I have a sample of 26 individuals. Those were requested to answer a questionnaire. Based on the collected info I have arranged my sample in 3 different groups: one with 16 individuals - GROUP A, ...
0
votes
0answers
44 views

Monte-Carlo Simulation Between Different Variables in the Same Model

I'd like to run a Monte Carlo simulation to test the null hypothesis that the predicted probabilities of two binary independent variables which are a part of my logistic regression model are equal ...
1
vote
0answers
14 views

How to test hypothesis with multiple 'or' restictions

I am seeking help to interpret a question and set up test procedures for a series of hypotheses (six in total, two of which I can answer). It is part of a weekly problem set we are given in class. The ...
0
votes
1answer
23 views

Do I need to correct for multiple testing if I perform one mediation model with 3 parallel mediators?

I am performing a mediation model with 3 parallel mediators, 1 independent variable and 1 dependent variable. My question is: is this considered a unique analysis, or should I control for multiple ...
0
votes
1answer
57 views

why correct for multiple comparisons in a correlation table?

I often read that in a correlation matrix the p-values for all correlations in a NxN matrix should be adjusted for multiple comparisons (for example: Multiple comparisons for correlation matrix?). ...
0
votes
0answers
13 views

In multiple restriction alternative hypothesis, should this be AND/OR or just OR?

Consider a null hypothesis under two restrictions: $H_0: \beta_1=u, \beta_2=v$ I encounter two ways to form the alternative hypothesis. One is: $H_A: \beta_1 \ne u $ $OR$ $\beta_2 \ne v (1)$ The ...
3
votes
0answers
93 views

A few questions regarding the practice of heterogeneous treatment effect analysis (a.k.a, interaction detection or subgroup analysis) methods

Imagine I am looking at a randomized experiment between a control and one or more treatment conditions. For example, I have a treatment that aims to get people out of debt. I randomize people to ...
0
votes
0answers
53 views

Using local FDR and q-values in practice?

I've learned that the local FDR and q-values are the estimated FDR given a fixed rejection region, which is the opposite of other frequentist methods where you fix the error rate first to estimate the ...
0
votes
0answers
23 views

Bayesian statistic in model comparison

I am trying to use the Bayesian statistic in model comparison. Let's say I have 4 different pharmacokinetic models which I use to fit breast MRI data. Each of this model M gives me a certain number of ...
2
votes
1answer
84 views

Multiple Test Correction: Can we divide dataset into subgroups?

Assume that the researcher wants to test huge amount of hypothesis and present his discoveries. He has data from two groups, eg, 100 people vs 100 people, and he wants to 1) having 20.000 genes ...
0
votes
1answer
42 views

Is it correct to do pre-filtering before multiple tests?

I often see that authors of large-scale studies that theoretically have to perform multiple test correction for enormous amount of tests do pre-filtering, in other words - throw away all tests that ...
0
votes
0answers
13 views

Statistical test to compare cellular types

I have a statistical question. I have 10 patients and 5 cellular lines. For each patient the gene expression of 6 genes was measured. Imagine the y-axis represents the expression of that genes in ...
0
votes
0answers
20 views

Using q-values in practice?

I have been trying to wrap my head around the concept of q-values for a while, but I'm fairly new to statistics so have been struggling with understanding papers. From Storey's paper, I'm given to an ...
2
votes
1answer
104 views

Comparing different treatments using logistic regression

I have some data on testing different pesticides on insects. I have applied different concentrations and different pesticides (total 11 treatments) for 24 hours, and have 6 different data points. I'm ...
0
votes
1answer
65 views

Adjust for multiple comparisons in several mixed models

I am fitting several generalized mixed models with binomial family to model proportions data(dependent variable) and 3 factors.The data looks like this: Sample Day ligand Condition Conc1 Conc2 .... ...
1
vote
0answers
110 views

How to test for conditional independence with mix of categorical and continuous data

I wish to test for conditional independence between $X$ and $Y$ given $Z$. However $X$ is continuous, while $Y$ and $Z$ are categorical variables. My idea is to test if $P(X|Z = z) = P(X|Y = y, Z = z)...
0
votes
0answers
158 views

How to find the power of a permutation test?

I am new to the permutation test and has found some online resources to learn how to conduct the permutation test. But I have yet to find a good explanation of how to find the power of a permutation ...
1
vote
0answers
22 views

How to choose the most effective treatment?

I randomly assign subjects to $N$ different group and let each group take a different treatment (e.g., dose of a medicine). The treatment is labeled as $T_i$, where $i=1,2,...,N$. The metric I care ...
2
votes
1answer
52 views

How Is This Design a Randomized Complete Block Design?

I recently started learning about one-way ANOVA, multiple comparisons, and contrasts. At the moment, I have an introductory-level knowledge of these concepts. I'm analysing an experiment that ...
0
votes
1answer
21 views

How to match more than two samples?

I'd like to compare the average values of some variables between 3 different groups. I should control for some covariates between the members of these 3 groups. For that, I decided to use matching ...
3
votes
0answers
23 views

Advice for conducting the same experiment several times?

I'm very new to Bayesian statistics, but I've found myself in a situation where they might be able to help. I'll be running the same experiment repeatedly and analyzing the data every few months. The ...
2
votes
0answers
199 views

Interpreting odds ratio of multiple comparisons from a logistic regression model (using R)

I recently ran a logistic regression on categorical data and ran a Tukey multiple comparisons post hoc analysis using the glht ...
0
votes
0answers
16 views

How can I create and analyze a composite score for multiple pre- and post-intervention variables?

I have a relatively small sample (~30) for an exercise intervention study. I have pre and post intervention data for three variables that are all related to cardiovascular health but are not on the ...
-1
votes
1answer
25 views

How to compare curves behavior?

I want to know how I can compare the "behavior of curves" and see what curves behave more similar. In the link below, in the 1st graph we have 4 curves through out time. http://www.sooty.ca/sooty/BDI....
0
votes
0answers
132 views

Adjustment for multiple tests for a mediation with multiple DVs? (Process/SPSS)

I hope my question has not been addressed in another post; I could not find any response to my issue. I'm conducting a mediation analysis with 1 IV (X), 1 mediator (M), and 4 DVs (Y). I use Process, ...
0
votes
0answers
5 views

how to determine the number of comparisons in multiple testing p value adjustment

I have a dataset of 100 biomarkers. For each biomarker, there are 3 groups. In my analysis, I did pairwise ttest between each group and generate p-values for each biomarker. And I did this for all ...
0
votes
1answer
23 views

Check that therapy is effective with FWER control

I have the following data: weights of 50 patient before consuming an anti-anorexia drug and after it. I need to check whether the therapy was effective, controlling FWER on given level. I know how to ...
0
votes
0answers
32 views

What is the correct way to compare different treatments at different concentrations?

I did an experiment where I am comparing control vs. three drug treatments, each at three different concentrations, and each concentration has 4 replicates. The outcome is signal intensity. I am ...