Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
21 views

Post-hoc analyses with a synthetic control?

When running a traditional hypothesis test, it's common to "break out" the result by sub-groups using various post-hoc methods that control for multiple comparisons, such as the False Discovery Rate. ...
4
votes
2answers
49 views

Best way to test that one mean is greater than all the others

Suppose I have $k$ samples of different sizes, each of a different univariate variable. I want to test the significance of the hypothesis that the population of $k_0$ has a mean greater than all of ...
0
votes
0answers
19 views

Bonferroni correction for multiple testing and overlapping groups

I have allele frequency observations from a large geographic region that can be logically subdivided into smaller areas in a number of different and meaningful ways. If I were to test these ...
3
votes
1answer
27 views

Possibility to run multiple chi-square tests on 2x5 table

I'm analyzing data in which I have a dependent variable with two possible outcomes (yes or no) and an independent variable with five emotion conditions, neutral, angry, sad, ashamed, and afraid. The ...
1
vote
0answers
9 views

fdr correction for omics data

I have a experiment design with 2 factors and I have 850 genes from this experiment. I want to look at individual genes before i go to multivariate dimension reduction . So I plan to fit a two way ...
0
votes
0answers
11 views

Multiple Comparisons for paired control and treatment groups

I want to conduct a post-hoc analysis on the impact of a treatment compared to the control Each control and treatment were selected to that the underline features are similar. For e.g. in the ...
0
votes
1answer
28 views

Pairwise non-parametric test for small sample sizes

I was asked to check a small dataset (3 replicates for 5 treatments, each) for significant outcome differences. I give an example: The variance within replicates in the data cannot be assumed ...
2
votes
1answer
21 views

Does the bonferonni correction change the outcome of the p-value or is it just changin what you accept as significant?

For some reason I can't seem to integrate the bonferonni corrected p adjustment into my multiple comparisons t test in r so I need to do each of my 3 tests individually. What I wondered was do I need ...
0
votes
0answers
22 views

How to correct for multiple post-hoc tests in multi-factor ANOVA

Let's for the sake of this question assume we did an ANOVA with three factors A (3 levels), B (4 levels), and C (5 levels). We get significant main effects for A and B. In consequence we do three ...
0
votes
0answers
13 views

Inquiry on comparing quadratic regression models

I am conducting a project in which I am analyzing the relationship between stream flow and algal biomass for a 28 day interval. I have 5 different flow treatments (ranging from low to high flows). I ...
0
votes
0answers
18 views

Multiple Comparison Correction for Linear Regressions with Dependent Variables

I could use some help clarifying the use of multiple comparison corrections for a series of linear regressions when some of the variables being tested are calculated from each other. Toy problem to ...
5
votes
3answers
142 views

Should I adjust p-values when investigating an ANOVA interaction?

I'm conducting an experiment to compare 4 types of headphones. I'm interested in finding out whether the headphones differ in their subjective quality. Thirty subjects listened to each pair of ...
0
votes
0answers
32 views

post hoc test for unequal sample size and variance

I have a data frame on which I am doing ANOVA using three fixed independent factors of which one is gender (two levels 'male' and 'female'), temperature (three levels: 1,2,3) and quality (two levels: ...
0
votes
2answers
31 views

Increasing the power by dropping points: can I do it?

I am repeating a test on a large amount of data and FDR-correcting the p-values afterwards for multiple testing. Yet I still do not have enough power. However, I feel like it is not necessary to test ...
2
votes
0answers
19 views

Multiple comparison of non-normal, heteroscedastic data. What test should I use?

I have a set of brain pathology data. These were obtained by counting certain parameters in the brain. Due to availability of human brains, the amount of cases vary a lot across the different groups ...
0
votes
1answer
25 views

(Non-parametric) test for similarity between three or more vectors of observations?

I have three vectors each of six observations. I imagine other situations will arrive in which I might have more than three vectors with other lengths. I want a statistical test to compare them. ...
0
votes
0answers
12 views

TukeyHSD for non-pairwise comparisons

Following this question: Pairwise comparisons I'm not sure if I can perform a Tukey Honest Significant Difference test on the data I have. I've thought Tukey's method was only valid for pairwise ...
1
vote
2answers
36 views

How to adjust for multiple comparisons with 2 groups where several outcomes were measured?

I have two groups, Controls and Patients, with 6 measurements for each such as speed, amplitude and so on. I'd like to examine whether these measures are different for the 2 groups. So firstly I'd ...
0
votes
1answer
43 views

two-way multiple comparisons in R

This is a subset of my dataset: ...
1
vote
0answers
26 views

Generating t statistics from p values - binomial test multiple comparisons

I am using the binomial test to generate a series of p values. However, my method of multiple comparisons correction that I wish to apply to this series requires that I also have the corresponding t ...
0
votes
1answer
25 views

Confusing LSMEANS results

I am having trouble interpreting results from a multiple comparison test. Here is the model I ran: ...
3
votes
3answers
69 views

How to find the right test?

I have a problem in medicine: Each patient can have several (3) symptoms (A,B,C) of a disease at the same time. For each symptomtype we can measure a score i.e. A=0.2, B=0.45, C=0.8. The aim is that ...
0
votes
1answer
73 views

How to compare stochastic algorithms, before and after modification

I would appreciate suggestions on how I could make some specific statistical claims about the efficacy of changes to an algorithm. My statistics background is pretty minimal (I'm working on it). I ...
0
votes
0answers
35 views

Need a method for determining variable groupings in R

I am using R and trying to group one of my variables into larger groups so they have credibility. I have been manually setting each factor of the variable as the reference level, looking at all other ...
1
vote
1answer
37 views

Adjustment of p-values for multiple-comparisons in lsmeans

Using R's lsmeans, I'm testing the pairwise comparisons among factor1 levels for each different combination of factor2 and factor3 levels: ...
1
vote
0answers
19 views

How to determine statistical differences among locations and years for continuous (non-normal) temperature data using R

I have three years of daily water temperature data from four different rivers. The temperature data are not normally distributed, so I assume I need to use a non-parametric test. I would like to ...
1
vote
1answer
36 views

What is the difference between nominal significance and genome wide significance?

I need to determine a reasonable sample size for my Inifinium 450k methylation array case-control experiment and wondered why epigenome-wide association scan (EWAS) studies typically use a P value ...
2
votes
2answers
41 views

Comparing prevalence among sites. Fisher.test and pairwise.t.test

I'm studying the prevalence of one parasite on a host from different localities. I do it by assigning presence ("1") or absence ("0") to each host sampled. After the sampling, I got something like ...
3
votes
0answers
19 views

The link between FWER and max/min statistic

I am currently looking at lecture slides about fMRI data and I have the following: Let $H_{0_i}$ be the hypothesis that there is no activation in voxel $i$ (a cube area in the brain, a 3d version of ...
3
votes
2answers
58 views

What exactly is multiple testing / Bonferroni Correction? Can I use it with a single AB test resulting in a p-value?

I'm having trouble drawing the link between a single hypothesis test and when you would apply the Bonferroni Correction to apply for multiple testing. The way I see it, you effectively just ...
1
vote
1answer
59 views

R, Pairwise comparison of random variable in mixed model

We measured temperatures of a pond repeatedly every day at each hour for a month at two different depths (i.e., top and bottom). We want to see if the temperatures at the top of the pond are ...
2
votes
0answers
43 views

MANOVA as protection against Type I errors

OK, so I have absolutely scoured the internet, and it appears that there is consensus (among people who really care about Type I error) that doing a MANOVA before doing multiple ANOVAs does nothing to ...
0
votes
0answers
13 views

Is there an accepted way to compare within all demographic categories (e.g., within males, within females, within whites, etc.) using MANOVA

I have three experimental groups (group A, group B, and group C), and gave all participants the same 50-question questionnaire. I then can run a MANOVA to determine if their responses (quantitative) ...
1
vote
1answer
33 views

Why does power reduce when increasing the number of post-hoc comparisons?

I'm performing a Dunn's post test comparison of my data after a Kruskal-Wallis test (4 groups, treated and control each one has drug and vehicle injection). The comparisons I've been doing are ...
0
votes
0answers
24 views

What regression analysis to use? IVs with two levels and a DV with two conditions?

I'm trying to figure out what the best regression test to use for my data. I have three predictor IVs each with two levels. I also have a DV values belonging to two different conditions (A & B). ...
0
votes
1answer
28 views

Is it common practice to correct for multiple comparisons with multiple ANOVAs?

I gave three groups of participants (say groups A, B, and C) different treatments, then gave all groups the same questionnaire. The questionnaire has ~50 questions. I then run ANOVA on each of the ...
1
vote
0answers
24 views

Effective number of independent tests, M(eff), to FDR adjusted p-values

I am trying to adjust a set of raw p-values with FDR method (BH). One of the assumptions in BH method is test independence. I know that some tests are clones and another ones are testing a very ...
1
vote
1answer
75 views

Interpreting GLM Interaction Contrasts in R (using glht)

I am trying to do a BACI analysis on count data, and I am having trouble interpreting the output from multcomp::glht. I don't understand how the contrast's coefficients are related to effect size ...
2
votes
1answer
32 views

What p-value threshold should I use for many-variable probit?

I have a probit with 7,000 observations and 120 dummy independent variables, of which Stata omits 31 because of low n and 2 because of collinearity. I also have a set of five dummies to make a time ...
0
votes
0answers
21 views

Comparing growth estimates of multigroup growth curve model

I have fitted a growth curve model to describe the trajectories of a variable in 4 groups across three assessment points. I am interested in doing some pair-wise comparisons of the slopes between some ...
1
vote
0answers
24 views

Test retest reliability

I tested subjects and repeated the test 2 times, so I did it 3 times in total. My question is: does anyone know how to evaluate the test-retest reliability? Is there another method (besides ...
4
votes
1answer
71 views

Difference or proportional

A report was claiming that older individuals were not appreciated at technology firms. They studied a dozen factory locations of a tech firm where employees could (anonymously) suggest process ...
1
vote
1answer
34 views

How should I correct for multiple comparisons?

For my dataset, I have two groups. For each group I would like to perform a one-sample t-test to compare the value against 0. I would then like to test a two-sample t-test to compare the mean of each ...
1
vote
0answers
26 views

Bonferroni Corrections for multiple control groups

We recently ran a marketing test at work where we are looking at the amount of customers that make a transaction at the store if we do not email them, Email them, email them AND send them a physical ...
0
votes
1answer
25 views

Testing proportion differences on multi sample/multi category data

My data is like this: I collected three groups of data randomly collected from three groups: A:(Age < 20), B:(Age 30-50) , C:(Age > 50) I evaluated the level of people smokes with three categories ...
0
votes
0answers
5 views

repeat measures, continuous/scale data, 3 dependent variables, 7 independent

Children played a comprehension game on an iPad and their attentional patterns were observed and coded for using video coding software. The duration of 7 different behaviors were coded for. Pupils ...
2
votes
0answers
37 views

Is there a consensus on adjusting alpha for multiple contrasts if the main effect is significant?

After talking to a couple of statisticians, reading some book sections, internet articles, and forums like this, I am still unclear about multiplicity adjustments of post hoc tests and contrasts. ...
0
votes
1answer
98 views

False Coverage Rate for confidence intervals of positive likelihood ratios of multiple dependent tests

I want to calculate confidence intervals of positive likelihood ratios of multiple dependent tests. To adjust the confidence level for the 'problem of multiple comparisons', I believe I should ...
1
vote
1answer
95 views

No significant tests when using Benjamini-Yekutieli multiple testing correction on millions of tests

I am using a univariate filter to reduce the number of features prior to applying a learning algorithm to a huge binary classification dataset (22510066 features x 500 examples). All the features are ...
3
votes
1answer
31 views

What is the best method to analyze this standard example data

The MASS package in R has 'anorexia' data set that compares the results of 3 treatments on weights of patients. Which statistical method is ideal to analyze this data set? There is one control (Cont) ...