Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

1
vote
0answers
9 views

Hypothesis testing for inaccurate sampling

My problem is as follows: Imagine a set of points, their positions have been recorded with a rubbish GPS, so are not precise and have a high error (assume no systematic error). Call this set $I$ ...
0
votes
0answers
10 views

Comparison Category Rating (Pairwise comparision) with SPSS [on hold]

I'am new with SPSS (worked with Excel before) but have to use SPSS for my bachelor thesis and got some problems. I did a Comparison Category Rating with multiple Persons (Playing 2 stimuli and the ...
2
votes
0answers
25 views

Sample size and multiple comparisons problem on A/B tests

I'm running an A/B test in which we have 4 different messages on a landing page (the current one and 3 treatments/variations). My task is to determine the sample size for this test, based on a 5% ...
0
votes
0answers
21 views

least square mean

Here I read that when comparing groups with different characteristics (age, percentage of males, body mass index) means should be corrected for such imbalances. These corrected means are called ...
-1
votes
0answers
17 views

comparing more than 2 groups of different size having binary values

I have 4 groups (Group 1, Group 2, Group 3, Group 4) that could be seen as binomial distributions. The groups have different sizes and the elements of each group can assume [0,1] values. Each group ...
0
votes
0answers
21 views

How to correct the means of a variable in 4 groups matlab

I compute the mean of the variable Y in 4 groups (A B C D) that differ for age, gender and body mass index (BMI). ...
1
vote
1answer
36 views

Correction for Multiple comparisons using Mann-Whitney U test

I am using Mann-Whitney U tests to compare groups of high and low relationship quality on five psychological outcomes. My data is non-normally distributed which is why I am using Mann-Whitney U tests ...
1
vote
0answers
31 views

Hypothesis testing with finite interval as null hypothesis

There exist two-tailed tests ($H_0: \mu = c$) and one-tailed tests ($H_0: \mu \leq c$ and $H_0: \mu \geq c$). I am interested in the case of the null hypothesis being an interval, that is $H^{ab}_0: ...
1
vote
2answers
57 views

Simultaneous tests for general linear hypothesis question

I have run a probit regression and am now trying to run post-hoc tests. I am trying to compare differences between a 3 level factor variable. I am confused about the difference between running a ...
1
vote
0answers
23 views

$P$-value correction for ANOVA on repeated simulation

First of all, I would like to ask you if you know good literature that explains the theoretical underpinnings of multiple testing. Although I think I understand pretty well the basic principles, I ...
8
votes
2answers
215 views

Is there a multiple testing problem when performing t-tests for multiple coeffcients in linear regression?

This question comes from a discussion on the recent post by @rvl It's all in the family; but do we include the in-laws too? Here's a common scenario that I've seen many times. A researcher runs a ...
9
votes
1answer
118 views

It's all in the family; but do we include the in-laws too?

Suppose I have an experiment with two or more factors. An overall ANOVA is constructed, and then we follow-up with two or more sets of post hoc tests, say multiple comparisons. My question is about ...
2
votes
1answer
40 views

Controlling FWER using minP/maxT methods

A common approach for controlling the FWER in genomics analysis is the so-called minP/maxT procedure. See, for example: Dudoit, S., Shaffer, J.P., and Boldrick, J.C. (2003). Multiple Hypothesis ...
0
votes
1answer
45 views

Permutation test, multiple comparison problem

I am not sure whether I can use a permutation test to avoid multiple comparison problem. I have run an experiment and collected participants' reactions times (reaction times is simply the speed at ...
0
votes
1answer
52 views

Finding all relationships from 500 independent variables to 400 dependent variables (time series)

I am interested in finding statistically significant relationships from a set of 500 independent variables X (actually about 25 variables + their 25 annual moving average x 10 monthly lags) to a set ...
4
votes
0answers
61 views

Cherry picking and low p-values

Let's say I run a lot of univariate OLS regression models, say 200,000, with 50 data points, then cherry pick the best one (highest r-square). If my $p$-value for this model is way less than ...
0
votes
0answers
7 views

Which variables characterizes the separation of speakers into two known groups

I have 40 patients for which 12 summary measures have been computed. Some of the measures are somewhat correlated in the data, but measure quantitatively different things, so I would like to keep ...
0
votes
0answers
12 views

Hypothesis test for 2 Weighted Means

I was trying to compare pairs of medical cost from several customized groups. The medical cost is adjusted by age and gender distribution in the sample. The resulting per member medical cost in each ...
0
votes
1answer
27 views

Sample Size for Multiple Testing

I want to calculate sample size for a classic multiple testing problem of multiple doses in clinical trials. I have 3 groups: "high dose", "low dose" and "placebo", and I wish to test if at least one ...
0
votes
1answer
23 views

Fisher's exact test and Bonferroni correction

I compared Survivability (live or die) using tibial Intraosseous (io), humerus io, tibial io, intravenous epinephrine for patients who have cardiac arrest. We had a control group that received no ...
1
vote
0answers
15 views

Multiple test correction procedures for multiple regression with dependent predictors

A researcher is employing OLS multiple regression to examine the independent effects (i.e., partial correlations) of a moderate (~8) number of theoretically-relevant predictors on some outcome. The ...
7
votes
2answers
129 views

Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?

If a team of researchers perform multiple (hypothesis) tests on a given data set, there is a volume of literature asserting that they should use some form of correction for multiple testing ...
1
vote
0answers
12 views

Testing the difference between multiple paired proportions with different sample sizes

Is there a hypothesis test similar to the paired t-test for testing if the difference between a number of paired proportion estimates is significantly different from zero? The data consists of 2 lists ...
2
votes
1answer
38 views

When to apply FDR procedure when doing many t-tests for many dependent variables?

I'm looking to run a bunch of t-tests, and I'm trying to figure out the appropriate time to apply an FDR correction. I have four conditions and am doing pairwise comparisons amongst these conditions, ...
0
votes
0answers
16 views

Aggregate Pseudo-Paired T-Tests

I would like to determine whether there is a significant difference between two different servers with regard to run-time. My data is split into subgroups each of which is a different Page. The same ...
2
votes
1answer
58 views

Bootstrapping gives me significance, without it not so much

For instance, in the table below, a GLM (ANOVA) Tukey posthoc bootstrapped 1000 times gives me a confidence interval like this. The columns are mean difference, bias, standard error, and the bounds of ...
0
votes
1answer
26 views

compare multiple algorithms in the same dataset

I want to do a comparison between multiple algorithms when they applied to the same dataset. For example, I have a set of texts in Arabic and another version translated to English. I did a sentiment ...
0
votes
0answers
10 views

Statistical significance of difference between samples given total success rate when test was applied by group

I am comparing treatments that select members to receive some email, and we are measuring the response to that email. The emails are similar, but are sent to a different group of members and a ...
0
votes
0answers
57 views

Comparison of 2 data sets with different sample sizes

I have two data sets that have different sample sizes and I am unsure on how to compare them. Dataset 1 has 51 participants but dataset 2 has only 20 participants. The answers to the questions are ...
0
votes
0answers
24 views

Comparing similarities and differences of time series data

how to compare the shape of two time series data. E.g. comparing fluctuations of two time series data. like, how to quantify whether one is more fluctuating than the other?
3
votes
1answer
135 views

Multiple comparisons after Kruskal Wallis using the FDR approach. How to compute P values (Dunn or Mann-Whitney)?

I want to allow users of GraphPad Prism to use the False Discovery Rate (FDR) approach to multiple comparisons after nonparametric one-way ANOVA (Kruskal-Wallis). The idea is first to compute an exact ...
1
vote
0answers
25 views

Why conduct large scale multiple comparisons rather than multiple regression?

Is there ever a compelling reason to conduct multiple hypothesis tests versus a multiple regression when comparing a response in two treatment groups among many (possibly thousands or millions) ...
0
votes
0answers
12 views

Analysis of pairwise differences in species composition between sampling dates

I have six timepoints (months) with 60 samples each. I want to assess the species composition changes over time. For single taxa i used glht() (multcomp) with the ...
0
votes
2answers
62 views

Multiple testing adjustment in linear models

I have troubles with multiple testing correction (For example, bonferroni adjustment. I'm aware that it's not most correct one). ...
2
votes
0answers
41 views

Paired Hypothesis Test in TWO Linear Regressions

I am wondering how to perform a paired hypothesis test in a panel regression. I have been looking in many stats books but I have not found anything. The question goes: assume for $i=1,2,\cdots, N$ ...
0
votes
0answers
22 views

Biostatistics: testing for differences in protein composition in two tissues measured longitudinally

Thank you for your help! I am confounded. I am trying to analyze experimental research data that looks like this: Dependent variable (DV) components: Two populations of cells (GFP and LacZ). Each ...
0
votes
0answers
24 views

Multiple testing correction for 2x2 tables in SPSS

Hi I have 1 binary dependent variable (presence or absence of coeliac disease) and multiple binary independent variables (presence or absence of different symptoms). If I perform a chi square in SPSS ...
0
votes
0answers
23 views

Is it statistically admissible to run multiple iterations of the same model but change the base level factor in order to get specific comparisons?

I have an agricultural experiment in which I was interested in seeing how parental identity of dam and sire impacted the number of buds formed and the resulting number of buds that survived ...
3
votes
0answers
38 views

Methods for meta-analysis: z-transform (Stouffer's approach) vs. Fisher's method vs… infinite others? [duplicate]

Fisher's method and Stouffer's approach (z-transform) for meta-analysis roughly follow the same scheme: take the p-values from some different experiments on the same hypotheses apply to them some ...
3
votes
1answer
134 views

Statistical significance test for averages of correlation coefficients

My setting looks like that I have several subjects $i$ (let's say 5) and for each subject I measure the lists/vectors $A_i$, $B_i$, $C_i$ and $D_i$. Then, for each subject $i$, I calculate the ...
0
votes
0answers
8 views

Correcting error for multiple correlations

I have been given a dataset to look at, which is made up of lots of clinical assessments. We have one main measure we are interested in (looking at sleep), which has 9 subscales. Then there are a ...
0
votes
0answers
130 views

Forcing multcomp:glht() to calculate as many meaningful pairwise factor combination as possible

I have a very imbalanced dataset consisting of several habitats (i.e. "woodland", "grassland", "swamp", and so on) with different sample sizes (10 - 40) and species counts as outcome. Species counts ...
0
votes
0answers
40 views

Post-hoc analyses with a synthetic control?

When running a traditional hypothesis test, it's common to "break out" the result by sub-groups using various post-hoc methods that control for multiple comparisons, such as the False Discovery Rate. ...
4
votes
2answers
53 views

Best way to test that one mean is greater than all the others

Suppose I have $k$ samples of different sizes, each of a different univariate variable. I want to test the significance of the hypothesis that the population of $k_0$ has a mean greater than all of ...
0
votes
0answers
24 views

Bonferroni correction for multiple testing and overlapping groups

I have allele frequency observations from a large geographic region that can be logically subdivided into smaller areas in a number of different and meaningful ways. If I were to test these ...
3
votes
1answer
41 views

Possibility to run multiple chi-square tests on 2x5 table

I'm analyzing data in which I have a dependent variable with two possible outcomes (yes or no) and an independent variable with five emotion conditions, neutral, angry, sad, ashamed, and afraid. The ...
1
vote
0answers
18 views

fdr correction for omics data

I have a experiment design with 2 factors and I have 850 genes from this experiment. I want to look at individual genes before i go to multivariate dimension reduction . So I plan to fit a two way ...
0
votes
0answers
24 views

Multiple Comparisons for paired control and treatment groups

I want to conduct a post-hoc analysis on the impact of a treatment compared to the control Each control and treatment were selected to that the underline features are similar. For e.g. in the ...
0
votes
1answer
43 views

Pairwise non-parametric test for small sample sizes

I was asked to check a small dataset (3 replicates for 5 treatments, each) for significant outcome differences. I give an example: The variance within replicates in the data cannot be assumed ...
2
votes
1answer
31 views

Does the bonferonni correction change the outcome of the p-value or is it just changin what you accept as significant?

For some reason I can't seem to integrate the bonferonni corrected p adjustment into my multiple comparisons t test in r so I need to do each of my 3 tests individually. What I wondered was do I need ...