Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [multiple-comparisons]

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
19 views

How to compare three groups of data each with a different sample size

I have a sample of 26 individuals. Those were requested to answer a questionnaire. Based on the collected info I have arranged my sample in 3 different groups: one with 16 individuals - GROUP A, ...
0
votes
0answers
42 views

Monte-Carlo Simulation Between Different Variables in the Same Model

I'd like to run a Monte Carlo simulation to test the null hypothesis that the predicted probabilities of two binary independent variables which are a part of my logistic regression model are equal ...
1
vote
0answers
14 views

How to test hypothesis with multiple 'or' restictions

I am seeking help to interpret a question and set up test procedures for a series of hypotheses (six in total, two of which I can answer). It is part of a weekly problem set we are given in class. The ...
0
votes
1answer
19 views

Do I need to correct for multiple testing if I perform one mediation model with 3 parallel mediators?

I am performing a mediation model with 3 parallel mediators, 1 independent variable and 1 dependent variable. My question is: is this considered a unique analysis, or should I control for multiple ...
0
votes
1answer
47 views

why correct for multiple comparisons in a correlation table?

I often read that in a correlation matrix the p-values for all correlations in a NxN matrix should be adjusted for multiple comparisons (for example: Multiple comparisons for correlation matrix?). ...
0
votes
0answers
12 views

In multiple restriction alternative hypothesis, should this be AND/OR or just OR?

Consider a null hypothesis under two restrictions: $H_0: \beta_1=u, \beta_2=v$ I encounter two ways to form the alternative hypothesis. One is: $H_A: \beta_1 \ne u $ $OR$ $\beta_2 \ne v (1)$ The ...
3
votes
0answers
80 views

A few questions regarding the practice of heterogeneous treatment effect analysis (a.k.a, interaction detection or subgroup analysis) methods

Imagine I am looking at a randomized experiment between a control and one or more treatment conditions. For example, I have a treatment that aims to get people out of debt. I randomize people to ...
0
votes
0answers
43 views

Using local FDR and q-values in practice?

I've learned that the local FDR and q-values are the estimated FDR given a fixed rejection region, which is the opposite of other frequentist methods where you fix the error rate first to estimate the ...
0
votes
0answers
23 views

Bayesian statistic in model comparison

I am trying to use the Bayesian statistic in model comparison. Let's say I have 4 different pharmacokinetic models which I use to fit breast MRI data. Each of this model M gives me a certain number of ...
1
vote
1answer
67 views

Multiple Test Correction: Can we divide dataset into subgroups?

Assume that the researcher wants to test huge amount of hypothesis and present his discoveries. He has data from two groups, eg, 100 people vs 100 people, and he wants to 1) having 20.000 genes ...
0
votes
1answer
33 views

Is it correct to do pre-filtering before multiple tests?

I often see that authors of large-scale studies that theoretically have to perform multiple test correction for enormous amount of tests do pre-filtering, in other words - throw away all tests that ...
0
votes
0answers
13 views

Statistical test to compare cellular types

I have a statistical question. I have 10 patients and 5 cellular lines. For each patient the gene expression of 6 genes was measured. Imagine the y-axis represents the expression of that genes in ...
0
votes
0answers
19 views

Using q-values in practice?

I have been trying to wrap my head around the concept of q-values for a while, but I'm fairly new to statistics so have been struggling with understanding papers. From Storey's paper, I'm given to an ...
2
votes
1answer
93 views

Comparing different treatments using logistic regression

I have some data on testing different pesticides on insects. I have applied different concentrations and different pesticides (total 11 treatments) for 24 hours, and have 6 different data points. I'm ...
0
votes
1answer
46 views

Adjust for multiple comparisons in several mixed models

I am fitting several generalized mixed models with binomial family to model proportions data(dependent variable) and 3 factors.The data looks like this: Sample Day ligand Condition Conc1 Conc2 .... ...
1
vote
0answers
104 views

How to test for conditional independence with mix of categorical and continuous data

I wish to test for conditional independence between $X$ and $Y$ given $Z$. However $X$ is continuous, while $Y$ and $Z$ are categorical variables. My idea is to test if $P(X|Z = z) = P(X|Y = y, Z = z)...
0
votes
0answers
120 views

How to find the power of a permutation test?

I am new to the permutation test and has found some online resources to learn how to conduct the permutation test. But I have yet to find a good explanation of how to find the power of a permutation ...
1
vote
0answers
20 views

How to choose the most effective treatment?

I randomly assign subjects to $N$ different group and let each group take a different treatment (e.g., dose of a medicine). The treatment is labeled as $T_i$, where $i=1,2,...,N$. The metric I care ...
2
votes
1answer
37 views

How Is This Design a Randomized Complete Block Design?

I recently started learning about one-way ANOVA, multiple comparisons, and contrasts. At the moment, I have an introductory-level knowledge of these concepts. I'm analysing an experiment that ...
0
votes
1answer
19 views

How to match more than two samples?

I'd like to compare the average values of some variables between 3 different groups. I should control for some covariates between the members of these 3 groups. For that, I decided to use matching ...
3
votes
0answers
22 views

Advice for conducting the same experiment several times?

I'm very new to Bayesian statistics, but I've found myself in a situation where they might be able to help. I'll be running the same experiment repeatedly and analyzing the data every few months. The ...
1
vote
0answers
166 views

Interpreting odds ratio of multiple comparisons from a logistic regression model (using R)

I recently ran a logistic regression on categorical data and ran a Tukey multiple comparisons post hoc analysis using the glht ...
0
votes
0answers
13 views

How can I create and analyze a composite score for multiple pre- and post-intervention variables?

I have a relatively small sample (~30) for an exercise intervention study. I have pre and post intervention data for three variables that are all related to cardiovascular health but are not on the ...
-1
votes
1answer
25 views

How to compare curves behavior?

I want to know how I can compare the "behavior of curves" and see what curves behave more similar. In the link below, in the 1st graph we have 4 curves through out time. http://www.sooty.ca/sooty/BDI....
0
votes
0answers
107 views

Adjustment for multiple tests for a mediation with multiple DVs? (Process/SPSS)

I hope my question has not been addressed in another post; I could not find any response to my issue. I'm conducting a mediation analysis with 1 IV (X), 1 mediator (M), and 4 DVs (Y). I use Process, ...
0
votes
0answers
5 views

how to determine the number of comparisons in multiple testing p value adjustment

I have a dataset of 100 biomarkers. For each biomarker, there are 3 groups. In my analysis, I did pairwise ttest between each group and generate p-values for each biomarker. And I did this for all ...
0
votes
1answer
23 views

Check that therapy is effective with FWER control

I have the following data: weights of 50 patient before consuming an anti-anorexia drug and after it. I need to check whether the therapy was effective, controlling FWER on given level. I know how to ...
0
votes
0answers
30 views

What is the correct way to compare different treatments at different concentrations?

I did an experiment where I am comparing control vs. three drug treatments, each at three different concentrations, and each concentration has 4 replicates. The outcome is signal intensity. I am ...
1
vote
1answer
25 views

Implications of descriptive statistics on parameter estimates

I am working on my dissertation and a question dawned on me after I ran my analyses. I studied whether the interaction between marital conflict and gratitude on mental health differed between first-...
0
votes
0answers
32 views

How to use Bonferroni correction - correct by no of parameters? No of models per dependent variable?

I have a simple question about when corrections such as Bonferroni should be applied. Say I have a research question for which I have three separate dependent variables, and run three separate models ...
0
votes
1answer
107 views

How is SNK.test ( or others ) doing groups in R?

I used the SNK.test (Student Newman Keuls Test) to compare different factor: ...
1
vote
0answers
17 views

Formulating hypothesys on continuous and categorial variables independence

Consider following process. We have $k$ people. They commit actions and we write down time of these actions in common list. We want to test the hypothesis that time of events is independent with who ...
1
vote
1answer
38 views

Formal definition of multiple hypothesis testing/multiple comparisons problem

I've been doing research on the topic of multiple comparisons, and most of the definitions I came across with, had the words multiple simultaneous statistical tests in them. Can anybody define ...
0
votes
0answers
15 views

Can you generate a 3-group ANOVA with significant F-ratio and all pairwise comparisons non-significant AND other restrictions?

This question is based on other questions and on my work writing statistics questions for online homework delivery systems.  (Happy to provide more info on this if requested.) I recognize ...
0
votes
2answers
25 views

experiment with 3 arms: analyzing results when one of 3 pairwise tests shows significance

I have an experiment with 3 groups: group 1 got no email group 2 got an email with 10 dollars incentive group 3 got an email with 20 dollars incentive to convert to a specific product. Conversion ...
0
votes
0answers
33 views

single p value from multiple experiments using the Wilcoxon sign rank test

Suppose I perform five identical independent experiments. Each experiment compared values between pairs of the same matched samples. Does it make statistical sense to compute a single p value from ...
0
votes
0answers
17 views

Multiple comparisons of polynomial fits.

I have some data: X and Y. I fit it as a constant, linear, quadratic and cubic polynomial functions. After this I calculate sum of squares $SS_), SS_1, SS_2$ and $SS_3$, respectively. Now I want to ...
3
votes
2answers
106 views

Multiple comparisons correction with independent groups

A question: I have a study with 3 experiments, each involving a between-subjects comparison of a treatment vs. control group. The groups in each experiment are independent samples and no groups are ...
0
votes
0answers
51 views

Multiple comparison correction in LME model

I have a dataset that looks like this (but longer): ...
0
votes
0answers
27 views

Is search for NN hyper parameters a Multiple comparisons problem?

Does search for NN hyper parameters (number of layers, neurons, learning rate etc) in nested loops comes in some price like e.g. narrower model confidence intervals, over-fitting etc. If yes can it be ...
0
votes
3answers
76 views

Does this situation necessitate multiple comparison corrections?

I am looking to run multiple t-tests on a dataset but I am unsure if the types of tests I am running would necessitate a multiple comparison correction or not. Below are essentially the tests I will ...
3
votes
1answer
403 views

Negative binomial pairwise comparison in R

I have fitted a negative binomial regression model to my data, and the summary of this compares latency of 3 resources to that of burrows: ...
0
votes
0answers
18 views

Correcting for multiple comparisons using ANOVA and posthoc testing

I have a question about best practice in my dataset. It consists of 50 variables from 30 samples grouped in three groups. I want to find which variables are significantly different between these ...
0
votes
0answers
31 views

Do I need to “correct”?

I am testing whether a certain learning intervention affects student achievement. I have two groups of students (control and experimental) with 16 students each. I have their pre-test and post-test ...
0
votes
1answer
42 views

How to ı calculate sample size?

I am planning a study for comparing tear levels (which I will measure with the Schirmer test) between two groups. Group 1: persons with normal blood vitamin D levels and Group 2: persons with low ...
1
vote
1answer
1k views

Post Hoc Pairwise Comparison of Interaction in Mixed Effects (lmer) Model

I have searched other posts and textbooks and found numerous variations of the pairwise comparisions for my mixed effects model using code from the multicomp ...
0
votes
0answers
284 views

differences between pairwise.t.test and p.adjust

I have searched to see if already answered but couldn't find an exact response, but apologies if its a duplicate. I am also very new to R and programming so apologies if the code is messy and I am ...
0
votes
0answers
92 views

Post hoc of interaction by comparison to marginal means?

I have a student who has run a 3x7 mixed ANOVA and found a significant interaction. It seems to me that the appropriate way to post hoc this (in the context of this experiment) would be to compare ...
5
votes
2answers
271 views

Regression when output is in a specific interval

Say my output is continuous on the interval $[-1, 1]$. Is there a standard technique to deal with this? Do I just use logistic function, same as in logistic regression, but for actual regression ...
1
vote
0answers
481 views

NA's appearing in my logistic regression summary output. How should I procede?

I have been following multiple posts on this same topic and understand why I have NA's in my summary table. I am wondering what is an appropriate course of action in order to conduct a post hoc ...