Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
13 views

What regression analysis to use? IVs with two levels and a DV with two conditions?

I'm trying to figure out what the best regression test to use for my data. I have three predictor IVs each with two levels. I also have a DV values belonging to two different conditions (A & B). ...
0
votes
1answer
17 views

Is it common practice to correct for multiple comparisons with multiple ANOVAs?

I gave three groups of participants (say groups A, B, and C) different treatments, then gave all groups the same questionnaire. The questionnaire has ~50 questions. I then run ANOVA on each of the ...
1
vote
0answers
13 views

Effective number of independent tests, M(eff), to FDR adjusted p-values

I am trying to adjust a set of raw p-values with FDR method (BH). One of the assumptions in BH method is test independence. I know that some tests are clones and another ones are testing a very ...
1
vote
1answer
20 views

Interpreting GLM Interaction Contrasts in R (using glht)

I am trying to do a BACI analysis on count data, and I am having trouble interpreting the output from multcomp::glht. I don't understand how the contrast's coefficients are related to effect size ...
2
votes
1answer
24 views

What p-value threshold should I use for many-variable probit?

I have a probit with 7,000 observations and 120 dummy independent variables, of which Stata omits 31 because of low n and 2 because of collinearity. I also have a set of five dummies to make a time ...
0
votes
0answers
11 views

Comparing growth estimates of multigroup growth curve model

I have fitted a growth curve model to describe the trajectories of a variable in 4 groups across three assessment points. I am interested in doing some pair-wise comparisons of the slopes between some ...
1
vote
0answers
13 views

Test retest reliability

I tested subjects and repeated the test 2 times, so I did it 3 times in total. My question is: does anyone know how to evaluate the test-retest reliability? Is there another method (besides ...
4
votes
1answer
68 views

Difference or proportional

A report was claiming that older individuals were not appreciated at technology firms. They studied a dozen factory locations of a tech firm where employees could (anonymously) suggest process ...
1
vote
1answer
30 views

How should I correct for multiple comparisons?

For my dataset, I have two groups. For each group I would like to perform a one-sample t-test to compare the value against 0. I would then like to test a two-sample t-test to compare the mean of each ...
1
vote
0answers
22 views

Bonferroni Corrections for multiple control groups

We recently ran a marketing test at work where we are looking at the amount of customers that make a transaction at the store if we do not email them, Email them, email them AND send them a physical ...
0
votes
1answer
18 views

Testing proportion differences on multi sample/multi category data

My data is like this: I collected three groups of data randomly collected from three groups: A:(Age < 20), B:(Age 30-50) , C:(Age > 50) I evaluated the level of people smokes with three categories ...
0
votes
0answers
4 views

repeat measures, continuous/scale data, 3 dependent variables, 7 independent

Children played a comprehension game on an iPad and their attentional patterns were observed and coded for using video coding software. The duration of 7 different behaviors were coded for. Pupils ...
2
votes
0answers
20 views

Is there a consensus on adjusting alpha for multiple contrasts if the main effect is significant?

After talking to a couple of statisticians, reading some book sections, internet articles, and forums like this, I am still unclear about multiplicity adjustments of post hoc tests and contrasts. ...
0
votes
1answer
78 views

False Coverage Rate for confidence intervals of positive likelihood ratios of multiple dependent tests

I want to calculate confidence intervals of positive likelihood ratios of multiple dependent tests. To adjust the confidence level for the 'problem of multiple comparisons', I believe I should ...
1
vote
1answer
80 views

No significant tests when using Benjamini-Yekutieli multiple testing correction on millions of tests

I am using a univariate filter to reduce the number of features prior to applying a learning algorithm to a huge binary classification dataset (22510066 features x 500 examples). All the features are ...
3
votes
1answer
29 views

What is the best method to analyze this standard example data

The MASS package in R has 'anorexia' data set that compares the results of 3 treatments on weights of patients. Which statistical method is ideal to analyze this data set? There is one control (Cont) ...
0
votes
0answers
32 views

EMMEANS Pairwise Comparison OR Post Hoc Multiple Comparison?

I was running a 2-way Univariate ANOVA, and there are two tables in the results that compair the 3 levels of one of my IVs (Transformation_Model). Under Estimated Marginal Means, the table is titled ...
0
votes
0answers
16 views

How to find significance when performing many significance tests

I have 4 Likert scales in both a pre- and post-survey. Each scale shows significant change from pre to post. I'd like to identify which individual items have a significant change as well. There are ...
1
vote
0answers
22 views

Appropriate test for similarity of four vectors?

I have four samples under four treatment conditions, and I want to test the hypothesis that they increase in such a way that the relative differences between them stay the same. Pearson correlation ...
0
votes
0answers
34 views

Second Opinion(s) Requested: Doing 6 A/B Tests on a website with confidence levels around .8-92 is a bad idea, right?

A consultant recently used 6 A/B tests to compare the effects of various modifications to a website. The website is designed to prompt visitors to respond with an action. As a "control" for each test, ...
1
vote
0answers
28 views

Correction of P value for correlation matrix

If one has to perform correlation matrix, correction for multiple testing is likely to be useful. If significant P value is taken as (0.05/number_of_variables) (e.g. for 50 variables, 0.05/50 = ...
2
votes
1answer
37 views

How to estimate probability of $\geq$ n type I error in multiple testing comparison?

My question is about the calculation of the probability of making $\geq n$ type I error when $p$ independent statistical tests are made. I can calculate the probability of $\geq 1$ type I error with ...
0
votes
0answers
16 views

Little mess with confidence intervals for Holm-Bonferroni

I'm a newcomer in the problem of multiple comparisons. I wonder how to correct confidence intervals for Holm-Bonferroni method? I know that for Bonferroni method we can just change the confidence ...
0
votes
0answers
26 views

Are the statistical words contrasts and post hoc synonymous?

I have implemented a linear mixed-effects model using lmer{} model in R and used glht to to ...
1
vote
0answers
23 views

Bonferroni adjustment according to which families?

We have 17 subjects in a crossover design of three groups: Challenge agent (CA) CA+ compound of interest (CI) CI - Our data proves this has no physiological effect, and is thus not used in ...
0
votes
0answers
10 views

How to determine values of coefficients for a comparsion using factorial design?

I think my problem is best answered by answering the following example but the more general the answer and explanation the better: Given 2 factors X and Y, x with 2 levels x1 x2 and Y with three ...
3
votes
1answer
28 views
1
vote
1answer
27 views

Bonferroni method for multiple comparisons with 1-way layout

My statistics book describes the Bonferroni method of doing multiple comparisons with a one-way layout as essentially performing a t-test with each pair of interest with a significance level scaled ...
0
votes
0answers
13 views

False discovery rate from non-parametric tests with small n

I have a bunch of paired readings of bacterial abundance, sampled from two different places in the same patient and divided by total bacterial cells found in each location. Thousands of species were ...
1
vote
1answer
25 views

Unequal sample size one way ANOVA

I want to compare the means of X between BMI groups, but have unequal sample sizes between my groups. ...
13
votes
4answers
491 views

Confusion with false discovery rate and multiple testing

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can ...
0
votes
0answers
23 views

Optimal model/statistical test for my design?

I have a design with 1 between-subject factor and 2 within-subject factors as independent variables and 5 dependent variables (longitudinal accelartion, lateral acc., response time, first conscious ...
0
votes
0answers
28 views

Pairwise comparisons with covariates

I have a simple ANCOVA model such that outcome ~ sex + group with outcome being continuous, ...
0
votes
0answers
34 views

Kruskal-Wallis post hoc comparisons different results in different software used

I'm having some trouble analyzing my data because when I use StatSoft Statistica and run a Kruskal-Wallis, I get one set of values for multiple comparisons, but when the same analysis is run in XLSTAT ...
0
votes
0answers
17 views

Dunnett for comparing multiple means to a specific value

In text-book statistics, one can use the one-sample t-test for comparing a population mean to a specific value. What would be the correct way to conduct a Dunnett's procedure of comparing multiple ...
3
votes
0answers
15 views

Multiple comparisons of parameters from non-linear regression

Hei, I want to compare parameters a_i and b_i estimated by nonlinear regression (y=a_i*x/(b_i+x)) for different data-sets (let's say 8 different data sets). I have calculated the non-linear ...
1
vote
0answers
31 views

Multiple t tests or an ANOVA?

I have low anxiety and high anxiety participants (21 in high anxiety and 20 In low). I measured anxiety and then categorised Pts according to this. They completed two memory tasks. In one memory ...
0
votes
0answers
53 views

Significant Friedman's test vs. non-significant multiple comparisons

I am currently running a randomized, placebo-controlled, cross-over design for a new type of intervention. This intervention is compared to a placebo and another more conventional type of treatment. ...
0
votes
0answers
37 views

compare muliptle probabilities

I have built three decision tree model to predict the response for three offers (One model per offer). I want to find the best offer for each customer based on the predicted probabilities. All offers ...
4
votes
2answers
190 views

Using multiple comparisons corrections in Fisher p-value framework

I am wary of stepping into the civil war between Fisher vs Neyman-Pearson interpretations of $p$-value (which has been well elucidated here and here), but I've been pondering a question that I keep ...
0
votes
0answers
48 views

P-values adjustment with known results of some hypotheses

I ran an experiment - within-subjects design; 2 scenarios (congruent and incongruent one). In each scenario I measured overall task completion time, geo. mean of task completion time and accuracy. ...
0
votes
0answers
22 views

Multiple testing confusion, what is significant?

I'm learning multiple hypothesis testing and this confuses the hell out of me. This is just a simple "are these distribution significantly different?" question. Data: Treatment: 100k random ...
0
votes
0answers
21 views

Which means difference test to use?

Currently, my data is count data and I am analyzing it in Proc Glimmix in SAS. My data focuses on how many times birds from two treatments performed stand, sit, forage, ect.... So it basically looks ...
0
votes
1answer
105 views

How to calculate critical values for Dunnett procedure given alpha, df1 and df2

I am looking for an R function that can calculate critical values for a Dunnett multiple comparison procedure for any family-wise error rate and degrees of freedoms. In short, I need similar ...
1
vote
0answers
45 views

Independent t test significant but post-hoc planned analysis not

I was analysing two sets of data. In most cases for the study post hoc planned comparisons identified significance (<0.05), however for one sample, this was not achieved. However an independent t ...
1
vote
0answers
65 views

How much confidence can we have in statistically-based ranking for “safe” international airlines? [closed]

Some cites for such ratings exist, but they are not meant to be predictive. For example AirlineRatings.com awards stars based on questions like Is the airline on the European Union (EU) Blacklist? ...
1
vote
1answer
17 views

How to Compare more than two Averages of Proportions or Means of Proportions

My experiment deals with $60$ subjects, one third of them belong to type A subjects, $i=1..20$, another third to type B, $i=21…40$ and finally type C, $i=41..60$. Each of these $60$ subjects from the ...
2
votes
2answers
90 views

Is it appropriate to do an ANOVA on a feature selected via inspecting PCA results?

I've been given a dataset consisting of 8 dimensional feature vectors for 4 classes of objects. I was asked to find the features that best distinguish the classes, and write up a short report. My ...
0
votes
0answers
19 views

Multiple comparisons with genetic * environment interactions

I am analysing the next models with logistic or linear regression: Y = G*E + confounders, where G is a genetic variable (SNP) and E is an environment variable. The problem is the next: I have 13 SNPs ...
8
votes
1answer
127 views

Plain language meaning of “dependent” and “independent” tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...