Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
1answer
28 views

Is there a concern for multiple comparisons when ranking instead of using significance?

Let's say I have many ANOVA tests on the same data using different factor variables each time. Then, I want to rank the F-statistics for all of those tests. Notice that I'm not interested in ...
0
votes
2answers
37 views

How to summarize R-squared of several regressions, one per subject

I will explain my question, I have made a study and I have 10 regression (one for each subject). I have a significance for each regression, but in some subjects the value of R-squared is 0.5 and in ...
0
votes
0answers
30 views

Multiple testing & model selection using AICc

I have a situation very similar to this post. I am finding the best-fit mixed effects model among a set of 5 candidate models for 7 different dependent variables (y1 to y7) using the same dataset of ...
1
vote
0answers
33 views

How to specify a significance test for intercept and slope between multiple latent groups

I have done a Latent Profile Analysis on 34 symptom-distress-items at beginning of psychotherapy treatment. I came up with 5 latent groups. The entropy of the model was 0.779 The average Latent ...
1
vote
0answers
18 views

Determining sample size using power analysis in multiple testing context

I am currently trying to determine the required sample size for an experiment. As I want to use the statistical power concept (for a t-test) to determine the required sample size, I want to specify a ...
0
votes
0answers
4 views

Compare Percentages - Pt 2

Based on earlier responses, I have further developed my testing methodology. The attached, if I have understood and applied responses correctly, gives me a z-score in the last column. Is it now just ...
0
votes
0answers
9 views

Log-rank test in R for file bmt

I am trying to find if there is any difference between the groups in file bmt in R package KMsurv using log-rank test. I have set H0: no difference in hazard rates between groups, and H1: there is a ...
0
votes
0answers
10 views

Non-parametric paired data: several Wilcoxon signed-rank tests? Other?

Scenario: I've taken blood samples from a group of patients (n = 15) and measured the expression of one marker on three cell populations within each sample (transitional B, naive B, memory B). I'm ...
1
vote
1answer
46 views

What statistical test allows comparison of the 3rd quartiles between groups?

I am using r and have ~400 observations divided into 3 groups. Group A contains 199 observations, while B contains 173 and C contains 24. I have generated a boxplot of their corresponding measures ...
1
vote
0answers
21 views

Multiple testing correction in a graph

I am doing some statistical tests over a graph. Namely, each subject in my study has a graph (composed of nodes and edges), where each edge has two different values (let's call them F and S). I am ...
1
vote
0answers
33 views

Multiple Hypothesis Testing with Categorical Variables and Alternatives

Suppose we have a variable $Y_i$ which is a binary categorical variable, and we have a variables: $X_{i,1}$, $X_{i,2}$, ....$X_{i,n}$ ($i=1,2,....N$ where $N$ is sample size, and $n$ is the number of ...
0
votes
0answers
11 views

Analyzing groups across time points in GraphPad

I have a dataset consisting of measurements of the level of 28 biomarkers for each of 25 people at 7 time points (4900 measurements in all, minus a few blanks). Each person is in a certain group ...
0
votes
0answers
17 views

Pairwise test and random effect for Beta regression

I have percentage areas values from 4 treatments (25 replicates per treatment). I would like to compare these percentages. I am supposed to use a beta regression, because my response variable is a ...
0
votes
1answer
12 views

Checking if an Error Type Occurred After Running A Hypotheis Test - Split (A/B) Test

When doing Split (A/B) Testing, or really any other Statistical Hypothesis test for that matter, there is always some chance of committing a Type 1 or Type 2 error. I am wondering if there are any ...
0
votes
0answers
29 views

Calculating Tukey HSD for a subset of comparisons - can I do this?

A colleague has about 50 samples, each with 2 conditions (treated and non-treated). The data is continuous and normal, and there are 3 data points in each sample-treatment combination. They'd like to ...
0
votes
0answers
15 views

Plotting errorbars adjusted for multiple comparisons

I have run a 1 way basic ANOVA on 4 categorical variables. Afterwards, I ran multiple comparisons on all 4 groups (6 comparisons) & adjusted using the Tukey-Kramer method. This was all done in ...
0
votes
0answers
24 views

How to see if 2 variables affect another variable?

I have a list of stocks with their % change, along with 2 other variables (Price Difference and Volume Difference). I would like to know how (if any) the variables Price Difference and Volume ...
0
votes
0answers
16 views

Interpreting Tukey HSD p-value After ANOVA + FDR correction

I'm currently analyzing a dataset where I have gene expression pathway activation scores across three groups: two different treatments and one control (so, basically a matrix of the form pathways x ...
1
vote
0answers
9 views

How to calculate FDR

This is quite a basic question, however I am having trouble. I have 10 samples. I have calculated the p value for 500 discrete elements within each of the 10 samples. The 500 elements are common to ...
0
votes
0answers
15 views

Comparing algorithms: runtime vs accuracy

I am comparing several algorithms with respect to their runtime and accuracy on several datasets. I follow [1] and compute the average rank using the accuracy and apply the Friedman and Nemenyi tests. ...
0
votes
1answer
19 views

Multivariate Regression vs T-test: and implication for multiple comparisons

I have a data frame with different outcome measures (DV1:4) for participants some partcipant with additional IVs 1:2. ...
0
votes
0answers
15 views

multicomparison treatment against 2 controls

i'm new in "r" and in statistics. I have some data that i would like to analize. Just to give you a background of my experiment: 1) i have 4 treatments (ifferent concentration of drugs) 2) i have 2 ...
0
votes
0answers
11 views

How many samples do I need to reliably apply the Bradley-Terry model?

I would like to use the Bradley-Terry model in order to build a ranking between objects, starting from a pairwise comparison test. However, I have almost 50 objects to be compared and the comparison ...
0
votes
0answers
10 views

Which analysis do I use for this experimental design?

Seeking feedback from those with a better quantitative mind than my own! Currently preparing my doctoral research proposal. I want to determine the impact that various levels of technology ...
0
votes
0answers
7 views

Corrections needed when performing multiple analyses on the same database? [duplicate]

I have access to data from various health-related surveys. Colleagues with a hypothesis in mind ask me to run their analyses. Even though each of them is unaware of the others' work, we are doing ...
0
votes
0answers
11 views

Multi-Model Inference - Post Hoc tests

Is there a way to do post hoc multicomparisons with Averaged Models of class "Average" gained from model.avg() of the MuMIn package in R?
2
votes
0answers
14 views

Degrees of freedom of a multiple comparission using estimability and lsmeans

I have a doubt about the degrees of freedom of a multiple comparission test. My data have the following structure: One data for each day, for 14 days, in each year season (auttum, winter, spring and ...
1
vote
1answer
21 views

When is it ok to NOT control for multiple comparison?

My initial feeling was never. But I notice that whenever people write up a larger study (or a meta-study) they do not adjust the significance thresholds of the individual measurements to account for ...
20
votes
3answers
2k views

Why aren't multiple hypothesis corrections applied to all experiments since the dawn of time?

We know that we must apply Benjamini Hochberg-like corrections for multiple hypothesis testing to experiments based on a single data set, in order to control the false discovery rate, otherwise all ...
0
votes
0answers
15 views

When to use MCP, or when to use closed testing procedure?

I am working as a programmer in a CRO and the pharmaceutical company tells us to carry out MCP test with 5 different contrasts. But I am not sure rationality of doing that without prior ANOVA ...
0
votes
0answers
28 views

Statistical method to combine/summarize multiple R squared values

I have a set of measurements using two different methods. I Have 6 samples that I have measurements for using both methods. I am trying to demonstrate how comparable the measurements are from the ...
2
votes
2answers
75 views

Incremental adjustments of $\alpha$ for multiple hypothesis testing?

Given a nominal significance level of $\alpha$ for a single hypothesis test, I understand that if you do multiple tests, $i = 1$ to $m$, then according to Bonferroni's correction, the significance ...
0
votes
1answer
30 views

Assumptions on Multiple Comparison - Bonferroni and others

I was asked for an alternative procedure to multiple comparisons which does not require independence of the multiple tests being applied. I´ve found some readings where Bonferroni´s correction is said ...
0
votes
0answers
18 views

How to compare contributory values over time? (with missing values)

Context I am working on a project considering computers. I have a dataset containing around 50 product characteristics for each computer. Also, for each computer, I have data about the product price ...
0
votes
2answers
22 views

Chi-squared for multiple samples

I have a set of data from an experiment where people are asked to remember items of different types; let's call them A, B and C, such that they can either recall them or not. Here is an example ...
0
votes
0answers
23 views

Multiple comparison hypothesis testing, combining p-values of subject data

I am thoroughly confused at the correct statistical procedure on a problem I have 7 different physical objects were ranked, subjectively, by 14 people on 6 different attributes (say soft/hard, ...
1
vote
1answer
129 views

What is the best way to determine which proteins are significantly bound on a testing chip?

I've got a question about the data from a biological experiment. Three times the same 1024 different proteins are spotted on one testing chip. Target of the experiment is to see whether certain ...
1
vote
1answer
32 views

How to apply FDR when you have two types of test statistics?

I'm testing the bivariate correlations among a large dataset of plant traits. Some of these correlations have highly non-normal residuals that are normal when log-transformed, but other correlations ...
4
votes
1answer
70 views

Which multiple comparison method to use for a lmer model: lsmeans or glht?

I'm analyzing a data set using a mixed effects model with one fixed effect (condition) and two random effects (participant due to the within subject design and pair). The model was generated with the ...
1
vote
1answer
25 views

Testing statistical significance of three percentages with ANOVA?

I want to compare the mortality rates of three groups. Can I use ANOVA? If so, how will I deal with the percentages?
3
votes
2answers
43 views

Is Bonferroni correction used for multiple levels, multiple factors, or both?

Bonferroni correction can help us correct our alpha values in multiple statistical testing. One aspect that I am confused on: Is Bonferroni correction used when we have multiple levels (Does this ...
9
votes
4answers
540 views

Why don't Bayesian methods require multiple testing corrections?

Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn't require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry About Multiple Comparisons, 2012. I don't ...
0
votes
0answers
12 views

Testing many proportions against each other…use logistic regression?

If I have an A/B/C/D/E/F/G test - i.e. I'm testing 5 test variations vs a control - what corrections do I need to make to be able to determine the best variation? To find winners vs the variation I ...
0
votes
0answers
9 views

Multiple comparisons tests for Generalized Regression in JMP

I'm trying to figure out a way to do post hoc contrasts/multiple comparisons test (i.e. Tukey HSD) after using the Generalized Regression personality of JMP's model builder. Does anyone know how to ...
1
vote
0answers
16 views

Simultaneous confidence bands around an arbitrary function with known point-wise variance

Given some estimated function $\hat{y_i} = \hat{f}(\mathbf{X_i})$, with a known pointwise variance $\widehat{var}(\hat{y_i})$, I want to calculate marginal effects $\frac{\partial ...
3
votes
0answers
57 views

Difference between one-way ANOVA and all pairwise comparisons with Bonferroni correction [duplicate]

If you have five groups and you wish to know if there is a significant difference between any of the group means, you would have to do 10 pairwise comparisons to test all possible pairs of means. You ...
0
votes
0answers
5 views

Compute average pairwise correlation Y(X1) vs Y (X2) within many nonoverlapping intervals of variable X

I am pretty sure someone already solved this problem and shared the code. Could you please help to find it? I need to calculate the pairwise correlation for the values of a discrete function Y(X) ...
0
votes
1answer
48 views

Correcting for multiple comparisons - same test on different data, vs different test on same data

I have a list of p-values and want to correct for multiple comparisons, i.e. control for the accumulation of false positives. In general, should I adjust my approach depending on whether the p-values ...
0
votes
0answers
16 views

Establishing a hypothesis involving three measures

I have 50 Training instances. A training instance is a particular simulation with specific parameters. I have 30 different policies. When I run the experiment with one training instance and one policy ...
1
vote
0answers
22 views

Correctly determine alpha for repeating statistical test in non-independent (interpolated) timeseries data

Problem I have timeseries of measurements of which I also have determined the standard deviation of the noise. My goal is to determine whether or not the measurement is significantly different from 0 ...