Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
5 views

Pairwise test and random effect for Beta regression

I have percentage areas values from 4 treatments (25 replicates per treatment). I would like to compare these percentages. I am supposed to use a beta regression, because my response variable is a ...
0
votes
1answer
8 views

Checking if an Error Type Occurred After Running A Hypotheis Test - Split (A/B) Test

When doing Split (A/B) Testing, or really any other Statistical Hypothesis test for that matter, there is always some chance of committing a Type 1 or Type 2 error. I am wondering if there are any ...
0
votes
0answers
14 views

Calculating Tukey HSD for a subset of comparisons - can I do this?

A colleague has about 50 samples, each with 2 conditions (treated and non-treated). The data is continuous and normal, and there are 3 data points in each sample-treatment combination. They'd like to ...
0
votes
0answers
10 views

Plotting errorbars adjusted for multiple comparisons

I have run a 1 way basic ANOVA on 4 categorical variables. Afterwards, I ran multiple comparisons on all 4 groups (6 comparisons) & adjusted using the Tukey-Kramer method. This was all done in ...
0
votes
0answers
21 views

How to see if 2 variables affect another variable?

I have a list of stocks with their % change, along with 2 other variables (Price Difference and Volume Difference). I would like to know how (if any) the variables Price Difference and Volume ...
0
votes
0answers
10 views

Interpreting Tukey HSD p-value After ANOVA + FDR correction

I'm currently analyzing a dataset where I have gene expression pathway activation scores across three groups: two different treatments and one control (so, basically a matrix of the form pathways x ...
1
vote
0answers
7 views

How to calculate FDR

This is quite a basic question, however I am having trouble. I have 10 samples. I have calculated the p value for 500 discrete elements within each of the 10 samples. The 500 elements are common to ...
0
votes
0answers
13 views

Comparing algorithms: runtime vs accuracy

I am comparing several algorithms with respect to their runtime and accuracy on several datasets. I follow [1] and compute the average rank using the accuracy and apply the Friedman and Nemenyi tests. ...
0
votes
1answer
17 views

Multivariate Regression vs T-test: and implication for multiple comparisons

I have a data frame with different outcome measures (DV1:4) for participants some partcipant with additional IVs 1:2. ...
0
votes
0answers
13 views

multicomparison treatment against 2 controls

i'm new in "r" and in statistics. I have some data that i would like to analize. Just to give you a background of my experiment: 1) i have 4 treatments (ifferent concentration of drugs) 2) i have 2 ...
0
votes
0answers
7 views

How many samples do I need to reliably apply the Bradley-Terry model?

I would like to use the Bradley-Terry model in order to build a ranking between objects, starting from a pairwise comparison test. However, I have almost 50 objects to be compared and the comparison ...
0
votes
0answers
9 views

Which analysis do I use for this experimental design?

Seeking feedback from those with a better quantitative mind than my own! Currently preparing my doctoral research proposal. I want to determine the impact that various levels of technology ...
0
votes
0answers
7 views

Corrections needed when performing multiple analyses on the same database? [duplicate]

I have access to data from various health-related surveys. Colleagues with a hypothesis in mind ask me to run their analyses. Even though each of them is unaware of the others' work, we are doing ...
0
votes
0answers
11 views

Multi-Model Inference - Post Hoc tests

Is there a way to do post hoc multicomparisons with Averaged Models of class "Average" gained from model.avg() of the MuMIn package in R?
2
votes
0answers
8 views

Degrees of freedom of a multiple comparission using estimability and lsmeans

I have a doubt about the degrees of freedom of a multiple comparission test. My data have the following structure: One data for each day, for 14 days, in each year season (auttum, winter, spring and ...
1
vote
1answer
19 views

When is it ok to NOT control for multiple comparison?

My initial feeling was never. But I notice that whenever people write up a larger study (or a meta-study) they do not adjust the significance thresholds of the individual measurements to account for ...
20
votes
3answers
2k views

Why aren't multiple hypothesis corrections applied to all experiments since the dawn of time?

We know that we must apply Benjamini Hochberg-like corrections for multiple hypothesis testing to experiments based on a single data set, in order to control the false discovery rate, otherwise all ...
0
votes
0answers
14 views

When to use MCP, or when to use closed testing procedure?

I am working as a programmer in a CRO and the pharmaceutical company tells us to carry out MCP test with 5 different contrasts. But I am not sure rationality of doing that without prior ANOVA ...
0
votes
0answers
21 views

Statistical method to combine/summarize multiple R squared values

I have a set of measurements using two different methods. I Have 6 samples that I have measurements for using both methods. I am trying to demonstrate how comparable the measurements are from the ...
1
vote
2answers
63 views

Incremental adjustments of $\alpha$ for multiple hypothesis testing?

Given a nominal significance level of $\alpha$ for a single hypothesis test, I understand that if you do multiple tests, $i = 1$ to $m$, then according to Bonferroni's correction, the significance ...
0
votes
1answer
22 views

Assumptions on Multiple Comparison - Bonferroni and others

I was asked for an alternative procedure to multiple comparisons which does not require independence of the multiple tests being applied. I´ve found some readings where Bonferroni´s correction is said ...
0
votes
0answers
17 views

How to compare contributory values over time? (with missing values)

Context I am working on a project considering computers. I have a dataset containing around 50 product characteristics for each computer. Also, for each computer, I have data about the product price ...
0
votes
2answers
16 views

Chi-squared for multiple samples

I have a set of data from an experiment where people are asked to remember items of different types; let's call them A, B and C, such that they can either recall them or not. Here is an example ...
0
votes
0answers
18 views

Multiple comparison hypothesis testing, combining p-values of subject data

I am thoroughly confused at the correct statistical procedure on a problem I have 7 different physical objects were ranked, subjectively, by 14 people on 6 different attributes (say soft/hard, ...
1
vote
1answer
125 views

What is the best way to determine which proteins are significantly bound on a testing chip?

I've got a question about the data from a biological experiment. Three times the same 1024 different proteins are spotted on one testing chip. Target of the experiment is to see whether certain ...
1
vote
1answer
28 views

How to apply FDR when you have two types of test statistics?

I'm testing the bivariate correlations among a large dataset of plant traits. Some of these correlations have highly non-normal residuals that are normal when log-transformed, but other correlations ...
4
votes
1answer
47 views

Which multiple comparison method to use for a lmer model: lsmeans or glht?

I'm analyzing a data set using a mixed effects model with one fixed effect (condition) and two random effects (participant due to the within subject design and pair). The model was generated with the ...
1
vote
1answer
21 views

Testing statistical significance of three percentages with ANOVA?

I want to compare the mortality rates of three groups. Can I use ANOVA? If so, how will I deal with the percentages?
3
votes
2answers
43 views

Is Bonferroni correction used for multiple levels, multiple factors, or both?

Bonferroni correction can help us correct our alpha values in multiple statistical testing. One aspect that I am confused on: Is Bonferroni correction used when we have multiple levels (Does this ...
10
votes
4answers
459 views

Why don't Bayesian methods require multiple testing corrections?

Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn't require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry About Multiple Comparisons, 2012. I don't ...
0
votes
0answers
8 views

Testing many proportions against each other…use logistic regression?

If I have an A/B/C/D/E/F/G test - i.e. I'm testing 5 test variations vs a control - what corrections do I need to make to be able to determine the best variation? To find winners vs the variation I ...
0
votes
0answers
7 views

Multiple comparisons tests for Generalized Regression in JMP

I'm trying to figure out a way to do post hoc contrasts/multiple comparisons test (i.e. Tukey HSD) after using the Generalized Regression personality of JMP's model builder. Does anyone know how to ...
1
vote
0answers
14 views

Simultaneous confidence bands around an arbitrary function with known point-wise variance

Given some estimated function $\hat{y_i} = \hat{f}(\mathbf{X_i})$, with a known pointwise variance $\widehat{var}(\hat{y_i})$, I want to calculate marginal effects $\frac{\partial ...
3
votes
0answers
57 views

Difference between one-way ANOVA and all pairwise comparisons with Bonferroni correction [duplicate]

If you have five groups and you wish to know if there is a significant difference between any of the group means, you would have to do 10 pairwise comparisons to test all possible pairs of means. You ...
0
votes
0answers
4 views

Compute average pairwise correlation Y(X1) vs Y (X2) within many nonoverlapping intervals of variable X

I am pretty sure someone already solved this problem and shared the code. Could you please help to find it? I need to calculate the pairwise correlation for the values of a discrete function Y(X) ...
0
votes
1answer
44 views

Correcting for multiple comparisons - same test on different data, vs different test on same data

I have a list of p-values and want to correct for multiple comparisons, i.e. control for the accumulation of false positives. In general, should I adjust my approach depending on whether the p-values ...
0
votes
0answers
15 views

Establishing a hypothesis involving three measures

I have 50 Training instances. A training instance is a particular simulation with specific parameters. I have 30 different policies. When I run the experiment with one training instance and one policy ...
1
vote
0answers
21 views

Correctly determine alpha for repeating statistical test in non-independent (interpolated) timeseries data

Problem I have timeseries of measurements of which I also have determined the standard deviation of the noise. My goal is to determine whether or not the measurement is significantly different from 0 ...
2
votes
0answers
42 views

Multiple testing correction of a two-sided test when the distribution is not symmetric

I am performing a large number of likelihood ratio tests for whether a value of a parameter (let's call it $p$) in my statistical model is smaller or greater than a particular value (1). The ...
0
votes
0answers
34 views

P-value adjustment for multiple (in)dependent regression analyses, when and how to do it?

I am testing the effects of various treatments (different compounds/toxins) at different doses on a test animal in a common multi-factorial X multi-level design. So my IVs are the Treatment (Toxin1, ...
0
votes
0answers
19 views

Multiple hypothesis correction - small n, large p?

I have some biological data (expensive), with only 3 replicates/"samples", measuring methylation level of 450,000+ loci. I am doing hypothesis testing comparing 3 samples of one group to 3 samples of ...
0
votes
0answers
15 views

Methodical questions about multiple group comparison and power

I want to compare 5 groups with different sample sizes (1 control, 4 patient groups) with regard to one independent variable. I want to compare all patient groups against one (larger) control group ...
3
votes
1answer
34 views

Comparing fitted parameters of power-law distributions

I estimated using maximum likelihood the parameters of power law distributions, I would like to compare the exponents fitted to data from different regions and also to data from the same region but ...
6
votes
0answers
63 views

Why are multiple simultaneous inferences problematic, but not multiple inferences across time? [duplicate]

It's often claimed that a good practice in statistics is to make adjustments for simultaneous inference to control so-called family-wise error rates. So for example if we did 100 tests some would say ...
4
votes
1answer
59 views

Do I need a Bonferroni correction?

My study has me running several regression analyses to investigate my hypothesis, and I am struggling with the multiple comparisons problem. I have three separate metabolites as predictor variables, ...
3
votes
1answer
43 views

Least conservative multiple testing correction

This is a general question regarding multiple testing when making confidence intervals. Suppose I have an unknown vector $$\mu = (\mu_1, \dots, \mu_p).$$ My goal is to estimate this vector, and ...
0
votes
0answers
20 views

ANOVA and post-hoc analysis preconditions

I know that before performing a t-test it's necessary to check if the data are nearly normal and that are not too skewed and also that we can relax the skewness condition as we collect more data. I ...
0
votes
1answer
11 views

A/B testing multiple comparsion

Following is the experimental design - There are two treatment group and one control. Need to conclude if treatment 1 is better than treatment 2. Is there a statistical test that helps in answering ...
0
votes
0answers
12 views

Best possible statistical approach to compare n pair of variables along a set of genomic intervals

I am dealing a bioinformatics problem and need help in approaching it in the right direction. I have a set of variables that represent different kinds of genomic data and are real valued. I am trying ...
0
votes
0answers
13 views

Coherence matrices

I'm analyzing ECoG data that were recorded during a working memory task. There are 6 different subjects and they all have about 100 trials. Each subject has different electrode coverage (so not in the ...