Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
1answer
14 views

How many hypotheses are needed when researching one category with multiple likert-scale question

I am currently researching two groups on their willingness with 5 questions on a 7-point Likert-scale. My hypotheses are: $H_0$: There is no difference in willingness between the two groups. $H_1$: ...
0
votes
0answers
14 views

Is there a non-parametric repeated measures test for replicated block data?

Following on from an earlier question, I've got another question about comparing machine learning classifiers on various datasets. From the response to that question, and also to this paper I ...
0
votes
0answers
13 views

Significance of multivariate models and correction for multiple comparisons

I have performed a multivariate binary classification using a number of features (or variables), I will call them features from sets (A), (B) and (C). I have calculated the P value of this ...
0
votes
0answers
11 views

One way ANOVA: post-hoc comparison “adding letters to graph” [migrated]

I am working on a project for one of my classes and cannot figure out how to obtain the letters for treatment comparisons after using the tukeyHSD post-hoc test. Is there a simple way to do this? ...
3
votes
1answer
37 views

p-value correction for multiple t-tests?

My dataset consists of $n$ genes, each of them described by a vector of expression values, $5$ for "healthy" individuals, and $5$ for "unhealthy" individuals. I am going to run $n$ t-tests (one for ...
1
vote
1answer
40 views

If I have to run 500 statistical tests, should I consider each case separately or find the “fittest” one technique?

Suppose I have to run 500 tests. For example, compare two groups with respect to 500 variables. If for a given variable the two samples are normally distributed, I could do a Student's independent two ...
0
votes
1answer
34 views

Adjusting for multiple Kruskal-Wallis tests

I've run 3 different machine learning algorithms on 10 different datasets, generating an accuracy on each one. My hypothesis is that two of the algorithms are consistently better than the third. I've ...
0
votes
0answers
16 views

Pairwise Wilcoxon test gives wrong results

I'm trying to perform a pairwise Wilcoxon test in R. When using Bonferroni p-adjustment I got only 1's and 0's. When changing p-adj to 'none', I get this: ...
0
votes
0answers
22 views

What is the relationship between degrees of freedom and number of comparisons in Nemenyi's post hoc test?

I have some data consisting of non-normally distributed samples drawn from one of several different groups. A Kruskal-Wallis 1-way ANOVA rejects the null hypothesis that the medians of the groups are ...
2
votes
0answers
35 views

Running sequential tests as data is collected - practical advice for researchers

I've observed that many experimental labs in my field don't do a power analysis before they start collecting data. In practice, it seems that many researchers collect data and periodically run ...
0
votes
1answer
41 views

confidence intervals best through direct calculation or R's confint funtion

I am interested in pairwise comparison of means, through the calcul of confidence intervals (CI) of the difference between two given sample's means. However, CI calculated directly from each sample, ...
3
votes
3answers
231 views

Do I need to correct for multiple hypotheses? p-values all go to one

I have two groups, C and A, where C is control. In both groups there are about 3,000 proteins. For each protein we acquire six measurements in both C and A (these are mass spectrometry intensities by ...
0
votes
2answers
56 views

multiple significance testing of large data

I am doing analysis using multiple significance testing between about 20 parameters of data with >8000 rows. Since the number is reasonably large, even minor differences are coming with highly ...
0
votes
1answer
15 views

Correction of mutliple paired samples t-test

In one group of subjects I have four parameters (for e.g. A, B, C, D) measured in two conditions. I want to know whether these 4 parameters differ between the conditions by doing paired samples ...
1
vote
0answers
30 views

Tukey HSD, insignificant p-value when the confidence interval does not contain 0?

I am using TukeyHSD to do multiple comparisons. I am seeing strange results for some of the pairs: the adjusted p-value is greater than the specified significance level but the corresponding ...
0
votes
0answers
22 views

Multiple comparisons of proportions using complex survey data using SAS

I am interested in doing multiple comparisons of proportions using complex survey data in SAS. I can get estimates using proc surveyfreq or proc crosstab (using SAS-callable SUDAAN), but am not aware ...
1
vote
0answers
53 views

Comparison and testing multinomial coefficients

Straight to the problem I want to test the hypothesis that two coefficients of multinomial regression model are equal between them. For example, considering a model where Y is 3 level categorical ...
0
votes
1answer
80 views

How do I interpret a variance ratio test in Stata?

I need help with interpreting output from a Variance Ratio Test in Stata. $f = 1.2988$ (my variance) $\text{Degrees of freedom} = 249, 493$ I have Ho: ratio = 1 as my null Ha: ratio != 1 Am I ...
1
vote
1answer
36 views

Adjust the p-values or not?

This started as a simple discussion where everyone thought they new the answer, and ended up with arguments and dozens of paper quotations. Thoughts appreciated. We have 10 samples: Control, A, B, C, ...
1
vote
1answer
102 views

Multiple testing in correlation analysis over time periods

I have one variable measured once per time interval (say, once per year), and another variable measured periodically (say, once per day). The periodic measurements are autocorrelated. I am interest ...
0
votes
0answers
16 views

Comparing the difference in proportions for 3 different groups

I conducted an experiment and just need some general guidance on where to go from here. I know I have to do some kind of binomial test on my data. Suppose I have the following data: ...
0
votes
0answers
24 views

Multiple Comparison of Standard Deviations

I have a question about comparing multiple standard deviations: To start with, let's say I have calculated two standard deviations (s1, s2) from a series of random samples. If I want to determine ...
1
vote
0answers
19 views

get p value based on observed F and critical F

When conducting multiple comparisons, I can calculate a critical value via whatever method is appropriate for the situation. If the adjustment method I need is a common one such as Bonferroni, I can ...
0
votes
0answers
24 views

Do we need to correct p-values when using multiple models?

I am currently studying epidemiology. I am writing a paper that is examining 11 disease states in one species. I am using binomial logistic regression to examine 2 possible predictors (age and ...
1
vote
0answers
46 views

Estimating and comparing pre-post treatment effects in multiple pairs of treatment-vs-control groups

I have a dataset from an experiment where the goal was to study in which plant ecotype a treatment with a certain bacteria had the greatest effect. Hence the data consists of a number of control and ...
0
votes
0answers
4 views

Null distribution of the maximum and FDR control

Suppose we have a series of test statistics $T_{1},...,T_{m}$ each corresponding to a null hypothesis $H_{0}^{j}: \theta_{j} = 0$. Let $H_{0}: \theta_{1} = \ldots = \theta_{m} = 0$ denote the ...
1
vote
2answers
101 views

Any other non-parametric alternative to Kruskal-Wallis?

It looks like Kruskal-Wallis is the standard nonparametric test for more than two groups. The problem is that it does not tell which groups are different, except that whether there exists significant ...
0
votes
0answers
29 views

Rank sums in the Kruskal-Wallis output

I use Stata to perform a Kruskal-Wallis test on four groups of data (not normally distributed), and the results are shown below: ...
3
votes
0answers
110 views

How to best approach ANOVA with multiple response variables that might be correlated and Bonferroni Correction

I need some help verifying a simple analysis. I have completed a simple two factor experiment of more or less equal sample sizes. Let's assume for the case of this question that the sample sizes are ...
0
votes
1answer
71 views

Which post-hoc is more valid for multiple comparison of an unbalanced lmer-model: lsm or mcp?

After doing a model comparison with my mixed lmer model, I have a model with three main effects, no interaction, say ...
0
votes
0answers
41 views

How do I report the result of an mcp-Tukey-test, i.e. how do I report it's z value?

After doing a model comparison with my mixed lmer model, I have a model with two main effects, no interaction, say signal ~ factor A + factor B + (1|subj). Factor ...
3
votes
5answers
171 views

What's wrong with Bonferroni adjustments

I had read an article has this title "What's wrong with Bonferroni adjustments?" for Thomas V Perneger He summarized saying that Bonferroni adjustment have, at best, limited applications in ...
4
votes
1answer
84 views

What statistical test is performed by summary(glht(model, linfct=mcp(factor=“Tukey”)), test=adjusted(type=“none”))?

I have data that I have fit using lme with the following structure (Subject is implemented as a random effect in order to account for multiple paired comparisons): ...
1
vote
0answers
21 views

Multiple testing, alpha, and familywise error

I am confused why it is that in multiple testing, the expected number of false positives is: $$ \alpha \times\text{number of tests}\tag{1} $$ (That is, for $\alpha = 0.05$, you would expect $5\%$ ...
4
votes
2answers
72 views

Extent of multiple testing correction

A bit of a strange question. In my fourth year biostatistics class today, we were discussing when and when not to use multiple testing correction, and the professor made an offhand comment. He asked ...
0
votes
2answers
50 views

Do I struggle with multiple comparisons?

I know that if there is significant difference in means among groups in ANOVA, then one can perfrom post-hoc test ( Tukey HSD or something similiar) to get to know within which groups the difference ...
2
votes
2answers
39 views

Multiple models compared over same data

My question boils down to if I really have to use a multiple comparison adjustment or not, and if so, how I can do this to retain as much power as possible. I have multiple models (say 5) that are ...
3
votes
4answers
569 views

About the Bonferroni correction

I have 50 variables in my dataset. I have correlated each variable against each variable, thus I have $49·50/2 = 1225$ unique correlations when no variable is tested against itself. Now, suppose I ...
1
vote
1answer
44 views

How to test if many samples are from the same distribution

For each day of the week I observe $n_i$ independently chosen values from some process. I would like to be able to answer the following two questions. Are the distributions from which the samples are ...
2
votes
2answers
92 views

Should corrections for multiple comparisons be used when evaluating independent experiments?

I understand why corrections for multiple comparisons (e.g., the Bonferroni correction) are used in some cases. For example, if I run a a single experiment and check significance of many factors, the ...
0
votes
1answer
39 views

Multiple testing

I am regressing different dependent variables using same set of predictors for all dependent variables such as ...
0
votes
0answers
20 views

How to test differences in multiple pairwise distances? With mixed model?

I want to compare two set of variables: X1 = All the pairwise distances between individuals from group A and group ref R X2 = ALL the pairwise distances between individuals from group B and group ...
9
votes
0answers
121 views

Should we address multiple comparisons adjustments when using confidence intervals?

Suppose we have a multiple comparisons scenario like such as post hoc inference on pairwise statistics, or a multiple regression, where we are making a total of $m$ comparisons. Suppose also, that we ...
0
votes
0answers
13 views

post-hoc multiple comparisons with chi squared using spss v. 22

I am carrying out an analysis using a 4$\times$2 crosstab. I found an overall significant difference but I would like to find if there are significant differences among the 4 groups. Is there a way ...
0
votes
0answers
20 views

column proportions comparisons for one variable

I have a one variable with three categories like marriage/divorced/widowed. The counts are: 40, 23, 7. I use SPSS in this case. I know how to perform chi_squared test for this, and it shows that the ...
0
votes
1answer
27 views

How to compare concentration ratios

I did a spectrometry measurement with the evaluation of the concentration of an element. For the element of interest I've two peaks (with normal distribution). $ C_1 ± \sigma_1 $ where $ \sigma_1 ...
0
votes
0answers
26 views

Multiple comparisons, multidimensional data, several groups

Could you please help me with such a problem: Let's assume I have multidimensional data that consists of 30 samples and 10 variables. This is a 30x10 matrix. I have three groups of equal size (10 ...
0
votes
0answers
20 views

FDR calculation on non-p-value probabilities

From a meta analysis, I have a distribution of probabilities that are not p-values (i.e., they are not uniformly distributed when H0 is true). I am using these probabilities for an intersection-union ...
0
votes
0answers
19 views

regression analysis for dataset with few predictors and few samples

I have a small pilot dataset containing experimental measurements for 12 samples, 4 numeric predictors and 1 numeric outcome variable. The goal is to obtain a first rough estimate on the extent to ...
3
votes
2answers
20 views

Multiple comparisons when investigating consistency of a device

I am trying to compare the performance of two devices designed to cool anaesthetised patients. This is a substudy analysis and the patient allocation between the groups is not even. Once cooling was ...