Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
5 views

Multiple groups, Different Items in each group, Find how many pairwise comparisons of the items within each category and across all categories

Question on finding the number of pairwise comparisons to perform within a group and across different groups. For example: If I wanted to find how many pairwise comparisons of the employees I ...
0
votes
0answers
17 views

Are multiple testing corrections necessary when validating against an already processed and published database?

Short version: Do I have to apply multiple testing corrections when analyzing a single variable (from a multidimensional dataset) I have previously found to be relevant and validated in the lab using ...
1
vote
3answers
59 views

Number of false positives after Bonferroni or FDR

Let's say I am doing 100,000 tests. If the tests are independent, at an alpha of 0.05, we would expect (on average) 5000 of these tests to be false positives. The Bonferroni method would penalize ...
0
votes
0answers
36 views

Adjusting p-values of multiple correlations

I do a correlation test in r with cor.test over two matrices, M and A, the final correlation matrix is C. The cor.test provides a p.value or estimate. How should I correct for multiple testing the p....
0
votes
0answers
14 views

multiple paired-sample t-test and Bonferroni Adjustment

I have 4 questionnaires: A,B,C,D. I also have another 4 items ($i_1,i_2.i_3,i_4$) that each measures some aspect of acceptability or feasibility of the questionnaires, which were answered following ...
0
votes
1answer
26 views

Bonferroni correction

I want to validate a questionnaire with three scales summing up to a total score. For for all tests I conducted, I tested the scales separately and additionally the total score (so 3+1 and that total ...
0
votes
0answers
13 views

Executing Permutation Test in R with Hypergeometric P Value

I'm new to these forums, but hoping you can help me with a question. I am using phyper in R to compute p values that assess the significance of overlapping gene sets, though from whole exome ...
1
vote
1answer
31 views

Correction for multiple t-tests: necessary in my case?

The analysis consists in comparing the means for 6 phonological constraints to the level of indifference, defined as 3.5. The same participants provided judgments for all the 6 constraints, which ...
0
votes
0answers
17 views

Multiple hypothesis testing for bivariate correlation

I am doing bivariate correlations (Pearson) comparing 140 different variables against each other. I was wondering if there is any risk of false positives (or that MHT has an effect here) in terms of ...
1
vote
0answers
14 views

Multiple testing correction for correlated test statistics

I have a matrix, the rows stand for the different diagnostic subgroups (6) and the columns stand for 4 different test statistics, making in total 24 values of test statistics. I should mention that ...
0
votes
0answers
12 views

How to test group differences when elements within a group are not independent?

I have 4 groups. Each group contains scores for items collected at different time stamps from a real-time system. Here is some dummy data: ...
0
votes
2answers
31 views

Is it good practice to adjust for multiple comparisons when performing different multivariable regression models?

I have five different outcomes and the same group of independent variables which have been used for the five different multivariable Cox regression models. The indipendent variables I have used in the ...
0
votes
1answer
24 views

Algorithm for ordered curves clustering

I am looking for some statistical solution to the problem of testing the similarity of curves. I am working with multiply time series (sort of survival curves). The curves are calculated as a separate ...
1
vote
1answer
18 views

multiple testing with BH correction

I want to do genome wide association study. I have about 1,500,000 SNPs, and 36 samples with 18 from tumor cells and 18 from normal lood cells. We make fisher test or chi square test(we admit fisher ...
2
votes
2answers
76 views

what to do when outliers are identified?

In one task that measures the times that participants take to respond to each item of a task some of the response times were considered outliers as they are more than 3SD above the mean of the ...
0
votes
2answers
37 views

What if Bonferroni correction is too conservative for 3 correlations on same group of people?

I use Pearson's correlation to calculate association between Test A and performance on 3 subtests of test B. If I use Bonferroni correction for multiple comparison then alpha is 0.05/3 . we found ...
1
vote
1answer
22 views

Can you break up a pairwise comparison into components, so that individual participants don't have to assess every possible comparison?

I'm trying to do a pairwise comparison for 40 items. However, this results in $\frac{40 \times 39}{2} = 780$ combinations. Obviously no-one is going to sit through a survey that long. Is there a way ...
0
votes
0answers
14 views

How do I calculate the probability of getting a particular effect size given multiple comparisons and no true association?

I have a hypothetical research scenario involving multiple comparisons and I want to try and prove my hunch right or wrong! Following a medical intervention, scores for 12 outcome measures are taken ...
1
vote
0answers
22 views

Adjusting p-values for multiple comparisons with correlated variables

I have three samples of size $N=10^6$ for the three responses $Y_1,Y_2,Y_3$ (or one sample of size $N$ for the random vector $\mathbf {Y}$). For each response the sample is i.i.d., even if the three ...
0
votes
0answers
18 views

Difference in performing two-way ANOVA and one-way ANOVA

I was asked by reviewers to retest my hypothesis that states A1 is better than A2, B1, and B2 (2x2 factorial design where A/B are two levels of the first factor and 1/2 are two levels of the second ...
0
votes
0answers
22 views

Basic question about within group comparison

I have a group of n subjects. Each subject has 3 measurements x, y, and z. 1-What kind of statistical analysis I should perform to see if those three measurements are significantly different or not? ...
0
votes
0answers
16 views

Why does this inequality about the false discovery proportion hold?

I am reading an article from Statistics in Medicine ("Tutorial in biostatistics: multiple hypothesis testing in genomics"). For my question it is only relevant to know that $V$ denote the number of ...
0
votes
1answer
34 views

what test to use for comparing count data across multiple groups

I have five groups of students and 40 in each. And we counted the number of references spelling mistakes in their homework. We would like to compare to see if there exists any groups differences in ...
0
votes
0answers
10 views

Multiple comparison in the hypothesis testing in different segmentations

I have a question regarding multiple comparison in different segmentations of the A/B testing data. Do i have to correct p value if the underlying data are different for each test or is it only ...
0
votes
0answers
22 views

How to select important features with multiple datasets?

There are several feature selection methods for dataset, but it is difficult for me to find methods for multiple datasets. The example of my datasets is like below. Sensors are features, and they are ...
0
votes
0answers
18 views

Pairwise comparison among variance

I would greatly appreciate any help on this! I'm doing a research where I need to compare variances (or standard deviation) of several samples (9 samples). I did some research and found the "Bartlett ...
5
votes
1answer
42 views

Is the Scheffé test of contrasts the “best-case” for post-hoc tests?

I am reading through Explaining Psychological Statistics, 4th Edition and am stumped after the following passage: Scheffé (1953) understood that the best anyone can do when creating a complex ...
1
vote
1answer
16 views

Is alpha error inflation a problem with multiple ordered logistic regressions?

Following problem: I'm doing a survey where my independent variables are: (1) the perceived attractiveness of a product on a 7-Likert-scale and (2) the willingness to invest in such a product as a ...
2
votes
2answers
52 views

How do I avoid multiple t-tests (or is it ok to use multiple t tests here?)

I've searched around CV for a bit, and have not found a good answer as to what test to use instead of multiple t-tests in my particular situation. I am interested in comparing mean size (of fish) ...
1
vote
0answers
35 views

How to interpret and report significant glmmADMB interaction term?

I am analyzing a data set using glmmADMB and determining significance of variables using likelihood ratio tests to compare models with and without a given variable. ...
0
votes
1answer
38 views

How to test multiple comparisons of means of variables in R

I have a multiple N (>20) variables. I want to test $H_0$ that in at least one pair of them one variable have significantly higher values then the other. I know that I can perform a bunch of $\frac{...
1
vote
1answer
67 views

Multiple comparisons when examining the results of unrelated trials

Imagine that you have a data set that is like the one below. (The data is all made up and is not specific to the problem.) ...
2
votes
1answer
18 views

Should one adjust analyses if interim analysis is planned for an outcome other than the primary outcome?

It is widely accepted that analyses of clinical trials have to be adjusted if interim analyses are planned in order to decide if early termination is necessary due to evidence for superiority of one ...
3
votes
1answer
69 views

Beginner question on dice roll statistical test

I have 2 cohorts of 10,000 people. I give each person in cohort 1 a die $a$, and ask them to roll it. I record the values as a tally. I then repeat this process with cohort 2 and give them die $b$. I ...
0
votes
0answers
20 views

Multiple testing correction with Benjamini-Hochberg with NaN values in the results

I have a list of p-values, and I want to correct them for multiple testing using Benjamini-Hochberg procedure. However, the list contains NaN/NA values. I know that ...
0
votes
0answers
5 views

the Univariate F statistics for each variable have the same rank order as the structure coefficients on the first CV

I have done a one-way MANOVA with 2 groups of 15 sample units each and 10 variables measured on each sampling unit, followed by separate ANOVAs and a CDA. I find that the Univariate F statistics for ...
3
votes
1answer
21 views

Only interested in select pairwise comparisons - Do I need to run an ANOVA first?

I have data with two biological factors. The first factor has 2 levels (say, Alive vs. Dead) and the second factor has 80 levels (protein of interest). I'm only interested in whether the mean value ...
0
votes
0answers
12 views

Control for multiple comparisons in an exploratory study?

I have an exploratory study where I investigate the relationship between several predictor variables (>10) and a few outcome variables (4). The large number of variables is not random but conceptually ...
0
votes
0answers
22 views

Comparing Subgroups vs Overall Relative Differences across Treatment

I have two very large unpaired control / variant groups, and a dependent variable that is the success/failure of a binomial, with a relatively small effect size between the treatment groups. If I ...
0
votes
0answers
22 views

How to interpret non significant F-test for main effect (genotype), but significant differences in the means of the genotypes?

I am analyzing an experimental data set based on the trait values under drought condition. The experiment was carried out on three separate drought environments. First, I have done a single ...
0
votes
0answers
22 views

When to use FDR?

I have a problem with the concept of false discovery rate. Assuming I have a panel of 50 anticancer drugs for which their IC50 values are reported for 30 cell lines. On the other hand, I have a ...
0
votes
0answers
10 views

comparisons of means within grouping factor (after two-way ANOVA)

Let's say in each of ten very different schools, one half of students is offered free lessons in physics and the other is not. After the next examination phase, the physics skills of the two groups ...
0
votes
1answer
37 views

Is there a concern for multiple comparisons when ranking instead of using significance?

Let's say I have many ANOVA tests on the same data using different factor variables each time. Then, I want to rank the F-statistics for all of those tests. Notice that I'm not interested in ...
0
votes
2answers
44 views

How to summarize R-squared of several regressions, one per subject

I will explain my question, I have made a study and I have 10 regression (one for each subject). I have a significance for each regression, but in some subjects the value of R-squared is 0.5 and in ...
0
votes
0answers
38 views

Multiple testing & model selection using AICc

I have a situation very similar to this post. I am finding the best-fit mixed effects model among a set of 5 candidate models for 7 different dependent variables (y1 to y7) using the same dataset of ...
1
vote
0answers
35 views

How to specify a significance test for intercept and slope between multiple latent groups

I have done a Latent Profile Analysis on 34 symptom-distress-items at beginning of psychotherapy treatment. I came up with 5 latent groups. The entropy of the model was 0.779 The average Latent ...
1
vote
0answers
26 views

Determining sample size using power analysis in multiple testing context

I am currently trying to determine the required sample size for an experiment. As I want to use the statistical power concept (for a t-test) to determine the required sample size, I want to specify a ...
0
votes
0answers
4 views

Compare Percentages - Pt 2

Based on earlier responses, I have further developed my testing methodology. The attached, if I have understood and applied responses correctly, gives me a z-score in the last column. Is it now just ...
0
votes
0answers
10 views

Log-rank test in R for file bmt

I am trying to find if there is any difference between the groups in file bmt in R package KMsurv using log-rank test. I have set H0: no difference in hazard rates between groups, and H1: there is a ...
0
votes
0answers
14 views

Non-parametric paired data: several Wilcoxon signed-rank tests? Other?

Scenario: I've taken blood samples from a group of patients (n = 15) and measured the expression of one marker on three cell populations within each sample (transitional B, naive B, memory B). I'm ...