Tagged Questions

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

1
vote
2answers
58 views

Any other non-parametric alternative to Kruskal-Wallis?

It looks like Kruskal-Wallis is the standard nonparametric test for more than two groups. The problem is that it does not tell which groups are different, except that whether there exists significant ...
0
votes
0answers
23 views

Rank sums in the Kruskal-Wallis output

I use Stata to perform a Kruskal-Wallis test on four groups of data (not normally distributed), and the results are shown below: ...
-2
votes
0answers
38 views

Did I meet my goal? [closed]

Based on below did I meet the goal of < 10 minutes H0 = Median 15 Ha=Median less than 15 results Count (N) =207 Count for test= 188 Median= 10 Wilcoxon Statisic =2517.50 p value (1-sides)=0.000
0
votes
0answers
11 views

how can i calculate square sum of all the products of the summation? [migrated]

Rule.  The square of a sum is equal to the sum of the squares of all the summands plus the sum of all the double products of the summands in twos: But I do not know how to calculate the square of ...
3
votes
0answers
76 views

How to best approach ANOVA with multiple response variables that might be correlated and Bonferroni Correction

I need some help verifying a simple analysis. I have completed a simple two factor experiment of more or less equal sample sizes. Let's assume for the case of this question that the sample sizes are ...
0
votes
1answer
31 views

Which post-hoc is more valid for multiple comparison of an unbalanced lmer-model: lsm or mcp?

After doing a model comparison with my mixed lmer model, I have a model with three main effects, no interaction, say ...
0
votes
0answers
19 views

How do I report the result of an mcp-Tukey-test, i.e. how do I report it's z value?

After doing a model comparison with my mixed lmer model, I have a model with two main effects, no interaction, say signal ~ factor A + factor B + (1|subj). Factor ...
3
votes
5answers
148 views

What's wrong with Bonferroni adjustments

I had read an article has this title "What's wrong with Bonferroni adjustments?" for Thomas V Perneger He summarized saying that Bonferroni adjustment have, at best, limited applications in ...
4
votes
1answer
42 views

What statistical test is performed by summary(glht(model, linfct=mcp(factor=“Tukey”)), test=adjusted(type=“none”))?

I have data that I have fit using lme with the following structure (Subject is implemented as a random effect in order to account for multiple paired comparisons): ...
1
vote
0answers
20 views

Multiple testing, alpha, and familywise error

I am confused why it is that in multiple testing, the expected number of false positives is: $$ \alpha \times\text{number of tests}\tag{1} $$ (That is, for $\alpha = 0.05$, you would expect $5\%$ ...
4
votes
2answers
60 views

Extent of multiple testing correction

A bit of a strange question. In my fourth year biostatistics class today, we were discussing when and when not to use multiple testing correction, and the professor made an offhand comment. He asked ...
0
votes
2answers
42 views

Do I struggle with multiple comparisons?

I know that if there is significant difference in means among groups in ANOVA, then one can perfrom post-hoc test ( Tukey HSD or something similiar) to get to know within which groups the difference ...
2
votes
2answers
37 views

Multiple models compared over same data

My question boils down to if I really have to use a multiple comparison adjustment or not, and if so, how I can do this to retain as much power as possible. I have multiple models (say 5) that are ...
3
votes
4answers
538 views

About the Bonferroni correction

I have 50 variables in my dataset. I have correlated each variable against each variable, thus I have $49·50/2 = 1225$ unique correlations when no variable is tested against itself. Now, suppose I ...
1
vote
1answer
42 views

How to test if many samples are from the same distribution

For each day of the week I observe $n_i$ independently chosen values from some process. I would like to be able to answer the following two questions. Are the distributions from which the samples are ...
2
votes
2answers
78 views

Should corrections for multiple comparisons be used when evaluating independent experiments?

I understand why corrections for multiple comparisons (e.g., the Bonferroni correction) are used in some cases. For example, if I run a a single experiment and check significance of many factors, the ...
0
votes
1answer
35 views

Multiple testing

I am regressing different dependent variables using same set of predictors for all dependent variables such as ...
0
votes
0answers
16 views

How to test differences in multiple pairwise distances? With mixed model?

I want to compare two set of variables: X1 = All the pairwise distances between individuals from group A and group ref R X2 = ALL the pairwise distances between individuals from group B and group ...
6
votes
0answers
93 views

Should we address multiple comparisons adjustments when using confidence intervals?

Suppose we have a multiple comparisons scenario like such as post hoc inference on pairwise statistics, or a multiple regression, where we are making a total of $m$ comparisons. Suppose also, that we ...
0
votes
0answers
8 views

post-hoc multiple comparisons with chi squared using spss v. 22

I am carrying out an analysis using a 4$\times$2 crosstab. I found an overall significant difference but I would like to find if there are significant differences among the 4 groups. Is there a way ...
0
votes
0answers
17 views

column proportions comparisons for one variable

I have a one variable with three categories like marriage/divorced/widowed. The counts are: 40, 23, 7. I use SPSS in this case. I know how to perform chi_squared test for this, and it shows that the ...
0
votes
1answer
25 views

How to compare concentration ratios

I did a spectrometry measurement with the evaluation of the concentration of an element. For the element of interest I've two peaks (with normal distribution). $ C_1 ± \sigma_1 $ where $ \sigma_1 ...
0
votes
0answers
25 views

Multiple comparisons, multidimensional data, several groups

Could you please help me with such a problem: Let's assume I have multidimensional data that consists of 30 samples and 10 variables. This is a 30x10 matrix. I have three groups of equal size (10 ...
0
votes
0answers
16 views

FDR calculation on non-p-value probabilities

From a meta analysis, I have a distribution of probabilities that are not p-values (i.e., they are not uniformly distributed when H0 is true). I am using these probabilities for an intersection-union ...
0
votes
0answers
17 views

regression analysis for dataset with few predictors and few samples

I have a small pilot dataset containing experimental measurements for 12 samples, 4 numeric predictors and 1 numeric outcome variable. The goal is to obtain a first rough estimate on the extent to ...
3
votes
2answers
20 views

Multiple comparisons when investigating consistency of a device

I am trying to compare the performance of two devices designed to cool anaesthetised patients. This is a substudy analysis and the patient allocation between the groups is not even. Once cooling was ...
0
votes
0answers
38 views

Correcting for multiple testing on non-independent sliding windows

Question Values taken from adjecent windows in a sliding windows are corrolated. If I calculate a p-value from each window, how can I correct for the fact I have tested many windows, given that the ...
1
vote
1answer
24 views

Distribution comparison across different treatments

if I have a table like this In which, site = different sites of measurement (tissue or organ) type = different types of virus And I want to ask something about the data 1.What test can I use to ...
4
votes
1answer
42 views

What is the intuition for dependence assumption in Benjamini and Hochberg (1995)?

For dealing with multiple testing, Benjamini and Hochberg (1995) assume that the tests are independent. Benjamini and Yekutieli (2001) show that (1) the 1995 result also holds under (I think) weaker ...
1
vote
0answers
28 views

Multiple comparisons on nested subsets of data for a non-parametric test

My problem: I'm using data sets where some are nested in a non-parametric test that expects independent data sets. How can I correct for that in a way that is reasonable? Or even better: What would ...
1
vote
1answer
30 views

Difference in difference model with 2 treatment groups

I have an experiment which has 2 treatment groups(effects) and a control group. Up until this point in my analysis, I've been carrying out a DID using a regression equation of the form: Y= γD_t + ...
1
vote
0answers
72 views

MHT: Pre-Selecting statistical tests without Bias

Summary: The last formula boxed in red (which is a modified log likelihood from logistic regression) is a special non-differentiable loss function that is adapted to contain a Bonferroni correction in ...
6
votes
2answers
141 views

The meaning of “positive dependency” as a condition to use the usual method for FDR control

Benjamini and Hochberg developed the first (and still most widely used, I think) method for controlling the false discovery rate (FDR). I want to start with a bunch of P values, each for a different ...
5
votes
2answers
100 views

Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple ...
1
vote
0answers
19 views

How to adjust for multiple comparisons with a large number of ANOVAs?

I have a complex experimental design: 3 groups, each of which come on 4 occassions and I obtain recordings from 250 skin electrodes. I am interested in knowing which of the electrodes provides a ...
2
votes
2answers
39 views

Multiple comparisons with t-test: one group with different variance

If I have 3 groups that I want to compare with a t-test (correcting for multiple comparisons afterwards) but one of the groups has different variance than the other two, should I use a Welch ...
0
votes
0answers
24 views

Comparing multiple groups to a control group

I've made several biochemistry experiments, and in which I compare several groups to a control group. I want to compare the results of 8 such experiments, so basically I have 8 measurements in each ...
3
votes
0answers
23 views

Compact letter displays with non-estimable cases

Compact letter displays are similar to grouping bars often used to display the results of a multiple-comparison procedure, but can be used more generally. You give a list of sample means (or other ...
0
votes
0answers
14 views

Iinterpretation of coefficients of multiple comparisons in a logistic regression model

I'm confused about how to interpret coefficients in relation to the reference categories. I have two variables, A and M. A is a 3-level variable and M is a 4-level variable. The reference category for ...
0
votes
0answers
37 views

Why are Dunnett's p-values smaller than those from two-sample t-tests?

Since Dunnett corrects for multiple comparisons I would assume the p-value would always be higher than a direct two-sample t-test. I came across a situation where this was not the case. Possible ...
1
vote
1answer
24 views

Combining data sets that have been adjusted for multiple testing

I have what I think is a simple question. Let say I have 5 data sets with with a variable number of tests (say 1-10K) and I control the FDR in each set to 1% using BH. If I combine these BH ...
0
votes
1answer
18 views

How to compare the nested models which each of them comes from diffrent dataset?

I have four nested models.Every of them learned from different data sets. now I want to compare these models together.normally people try to compute the F-satistics. But for my case, it's bit harder, ...
0
votes
1answer
108 views

When to correct p-values in multiple comparisons?

I'm afraid that related questions didn't answer mine. We evaluate the performances of >2 classifiers (machine learning). Our Null hypothesis is that performances do not differ. We perform parametric ...
2
votes
2answers
186 views

Comparing performances during a race between 2 groups, which test should I use?

Imagine I organized a race between 2 types of ants. Type A and B. At the end of the race, I have the ranking, which looks something like : ...
3
votes
1answer
320 views

Multiple comparisons in a non-parametric test

I am working with a data set that is non-parametric and has 12 treatments. I performed the Kruskal-Wallis test and got a significant $p$-value, and now I would like to conduct a multiple comparisons ...
0
votes
1answer
128 views

FDR (multiple testing correction) with skewed p-value distribution

I am confused with multiple comparisons adjustments. I have a $p$-values with lot of ones ( due to many scores in foreground are 0) from a fisher-exact test. I get some $p$-values which are ...
2
votes
1answer
85 views

When should I use FDR or Bonferroni in multiple comparisons?

I have the following situations : (1) Very large number of variables - say 100 variables and 200 samples (observations). Although level of each variable is just 2 or 3 or 4 max. I am doing linear ...
1
vote
2answers
58 views

Multiple comparison on subgroups and overall comparison

I am trying to work out is there is any association between occurrence or not of an event in around 500 individuals and their randomised binary grouping (intervention vs placebo). However I also want ...
0
votes
2answers
28 views

What are correlations among between test statistics in multiple comparison procedures

In the book Multiple Comparisons Using R Bretz et al write that their R package multcomp improves power above other multiple comparison procedures by "accounting for the structural correlations ...
1
vote
0answers
19 views

Location test of the ratios of random variables

I've been struggling with this problem for a while now, with not much progress. Any suggestions would be helpful! Suppose I have $X_1, .., X_5, Y_1, ..., Y_5, X'_1, ..., X'_5, Y'_1, ..., Y'_5$, i.e., ...