Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
10 views

Correcting for multiple testing on non-independent sliding windows

Question Values taken from adjecent windows in a sliding windows are corrolated. If I calculate a p-value from each window, how can I correct for the fact I have tested many windows, given that the ...
0
votes
1answer
14 views

Distribution comparison across different treatments

if I have a table like this In which, site = different sites of measurement (tissue or organ) type = different types of virus And I want to ask something about the data 1.What test can I use to ...
4
votes
1answer
34 views

What is the intuition for dependence assumption in Benjamini and Hochberg (1995)?

For dealing with multiple testing, Benjamini and Hochberg (1995) assume that the tests are independent. Benjamini and Yekutieli (2001) show that (1) the 1995 result also holds under (I think) weaker ...
1
vote
0answers
10 views

Multiple comparisons on nested subsets of data for a non-parametric test

My problem: I'm using data sets where some are nested in a non-parametric test that expects independent data sets. How can I correct for that in a way that is reasonable? Or even better: What would ...
1
vote
1answer
14 views

Difference in difference model with 2 treatment groups

I have an experiment which has 2 treatment groups(effects) and a control group. Up until this point in my analysis, I've been carrying out a DID using a regression equation of the form: Y= γD_t + ...
1
vote
0answers
65 views

MHT: Pre-Selecting statistical tests without Bias

Summary: The last formula boxed in red (which is a modified log likelihood from logistic regression) is a special non-differentiable loss function that is adapted to contain a Bonferroni correction in ...
3
votes
2answers
39 views
+100

The meaning of “positive dependency” as a condition to use the usual method for FDR control

Benjamini and Hochberg developed the first (and still most widely used, I think) method for controlling the false discovery rate (FDR). I want to start with a bunch of P values, each for a different ...
5
votes
2answers
98 views

Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple ...
1
vote
0answers
9 views

How to adjust for multiple comparisons with a large number of ANOVAs?

I have a complex experimental design: 3 groups, each of which come on 4 occassions and I obtain recordings from 250 skin electrodes. I am interested in knowing which of the electrodes provides a ...
2
votes
2answers
27 views

Multiple comparisons with t-test: one group with different variance

If I have 3 groups that I want to compare with a t-test (correcting for multiple comparisons afterwards) but one of the groups has different variance than the other two, should I use a Welch ...
0
votes
0answers
18 views

Comparing multiple groups to a control group

I've made several biochemistry experiments, and in which I compare several groups to a control group. I want to compare the results of 8 such experiments, so basically I have 8 measurements in each ...
3
votes
0answers
18 views

Compact letter displays with non-estimable cases

Compact letter displays are similar to grouping bars often used to display the results of a multiple-comparison procedure, but can be used more generally. You give a list of sample means (or other ...
0
votes
0answers
13 views

Iinterpretation of coefficients of multiple comparisons in a logistic regression model

I'm confused about how to interpret coefficients in relation to the reference categories. I have two variables, A and M. A is a 3-level variable and M is a 4-level variable. The reference category for ...
0
votes
0answers
19 views

Why are Dunnett's p-values smaller than those from two-sample t-tests?

Since Dunnett corrects for multiple comparisons I would assume the p-value would always be higher than a direct two-sample t-test. I came across a situation where this was not the case. Possible ...
1
vote
1answer
20 views

Combining data sets that have been adjusted for multiple testing

I have what I think is a simple question. Let say I have 5 data sets with with a variable number of tests (say 1-10K) and I control the FDR in each set to 1% using BH. If I combine these BH ...
0
votes
1answer
17 views

How to compare the nested models which each of them comes from diffrent dataset?

I have four nested models.Every of them learned from different data sets. now I want to compare these models together.normally people try to compute the F-satistics. But for my case, it's bit harder, ...
0
votes
1answer
78 views

When to correct p-values in multiple comparisons?

I'm afraid that related questions didn't answer mine. We evaluate the performances of >2 classifiers (machine learning). Our Null hypothesis is that performances do not differ. We perform parametric ...
2
votes
2answers
179 views

Comparing performances during a race between 2 groups, which test should I use?

Imagine I organized a race between 2 types of ants. Type A and B. At the end of the race, I have the ranking, which looks something like : ...
3
votes
1answer
61 views

Multiple comparisons in a non-parametric test

I am working with a data set that is non-parametric and has 12 treatments. I performed the Kruskal-Wallis test and got a significant $p$-value, and now I would like to conduct a multiple comparisons ...
0
votes
1answer
70 views

FDR (multiple testing correction) with skewed p-value distribution

I am confused with multiple comparisons adjustments. I have a $p$-values with lot of ones ( due to many scores in foreground are 0) from a fisher-exact test. I get some $p$-values which are ...
2
votes
1answer
62 views

When should I use FDR or Bonferroni in multiple comparisons?

I have the following situations : (1) Very large number of variables - say 100 variables and 200 samples (observations). Although level of each variable is just 2 or 3 or 4 max. I am doing linear ...
1
vote
2answers
44 views

Multiple comparison on subgroups and overall comparison

I am trying to work out is there is any association between occurrence or not of an event in around 500 individuals and their randomised binary grouping (intervention vs placebo). However I also want ...
0
votes
2answers
23 views

What are correlations among between test statistics in multiple comparison procedures

In the book Multiple Comparisons Using R Bretz et al write that their R package multcomp improves power above other multiple comparison procedures by "accounting for the structural correlations ...
1
vote
0answers
18 views

Location test of the ratios of random variables

I've been struggling with this problem for a while now, with not much progress. Any suggestions would be helpful! Suppose I have $X_1, .., X_5, Y_1, ..., Y_5, X'_1, ..., X'_5, Y'_1, ..., Y'_5$, i.e., ...
0
votes
0answers
16 views

Multiple testing correction when confirming a correlation with an extended dataset

Consider a given dataset on which I performed a set of independent tests on different (non-independent) variables, and obtained several p-values. I estimated the number of independent tests as being ...
0
votes
0answers
19 views

Which multiple comparisons correction should I use?

I'm studying structure-function relationships in neurons and mainly focusing on correlations between dendritic length and excitability. My question is: When searching for correlations, which ...
1
vote
2answers
79 views

Multiple testing: contingency table or logistic regression approach for survey?

I have a survey asking a person's gender (M/F), geographic region (West, East, Midwest, South) and a few other demographic variables along with two dozen questions in which they can disagree / ...
1
vote
0answers
11 views

Correcting for multiple comparisons using simulated distribution

To check if unilateral pairs (defined below*) are coordinated (i.e. move together) in a flock of N coordinated individuals, we generate hypothetical (null) distribution of a certain focal parameter of ...
2
votes
0answers
35 views

MANOVA vs repeated measues ANOVA example

I have the following situation (4 independent treatment groups, each group is tested for 3 dependent variables) and was wondering what would be the best to do: What is the best test to do? I have ...
5
votes
1answer
168 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
1
vote
0answers
78 views

Multiple Hypothesis testing correction on non-independent data

Consider a vector $X_i$ with $i \in [1,N]$. Let us say that you divide the vector into two $X_{i}'$ with $i \in [1,j]$ and $X_i''$ with $i \in [j+1, N]$ for several $j$; then for each pair of ...
0
votes
0answers
28 views

ANOVA, unbalanced designs, missing data, and multiple comparisons

I am having several problems with my dataset and how best to analyze it. I have measured a series of plant phenology characteristics (with a seperate model for each one - I do not want to combine them ...
0
votes
1answer
24 views

How to summarize multiple adjusted pvalue into a single measure?

Suppose I have a test with multiple hypotheses. For example, if I have X as a gene set (a set of symbols), and A, B, C pathways (sets of symbols), I am testing if A is enriched in X, B in X, C in X ...
0
votes
0answers
22 views

Levene's test for multiple comparisions

by doing ANOVA of biological data from four species I've find that my data have unequal variance between the species. Now I would like to determine which species are different in this character. Is it ...
3
votes
0answers
73 views

Multiple comparisons with interactions of mixed lmer model and how to report them

Assume I test a number of patients repeatedly over time to see how a certain treatment changes their skin conductance in response to a certain colour (cond) after 2 months, 4 months, ... etc. I test ...
0
votes
0answers
40 views

Multiple hypothesis testing or simply p-value sum?

For a given dataset I test simultaneously $m$ different hypothesis, obtaining a p-value for each. Each of these hypothesis correspond to test if the values $X_i$, a measured quantity from the dataser, ...
0
votes
0answers
16 views

Multivariate multisample equivalence test

I have multiple observations from n groups\treatments. How to test the equivalence of n groups using all the variables? ...
1
vote
0answers
14 views

Why does max-t methods use only the maximum of generated t-values?

I am wondering why max-t methods use the maximum of the t-values to generate the distribution. Does this not make it overly conservative compared to other methods, since the critical value will be ...
0
votes
1answer
31 views

Alternative to Friedman test for single treatments (no repetitions)

I'm looking for a non-parametric (!) alternative to the Friedman test. I have > 2 subjects and measured one quantity per subject. I want to know whether there is a significant difference between any ...
1
vote
0answers
18 views

Corrected multiple comparison: is the preceeding ANOVA necessary at all? [duplicate]

To compare multiple groups, one usually runs ANOVA. If ANOVA rejects the null hypothesis, one goes through the multiple comparisons. The multiple comparisons are necessary to establish which group is ...
0
votes
0answers
25 views

Identifying formulas from MANOVA in a research paper

I need to identify the formulas for computing the group differences with Bonferroni corrections and Cohen's d effect size in the following paper (chart on the top of page 6): Research Paper What ...
0
votes
1answer
23 views

Correction for multiple comparisons with multiple response variables

I'm trying to determine what constitutes a family of hypotheses when measuring multiple response variables. In particular I have an A/B type test where I'm comparing several marketing campaigns ...
4
votes
1answer
158 views

False discovery rate of multiple regressions models

I have 10 models analysing pupal developmental rates (DR) and developmental times (DT) depending on a number of factors: ...
3
votes
1answer
102 views

Machine learning classifiers big-O or complexity

To evaluate the performance a new classifier algorithm, I'm trying to compare the accuracy and the complexity (big-O in training and classifying). From Machine Learning: a review I get a complete ...
0
votes
0answers
16 views

Advice on how to analyze these data

In my experiment, I have 312 separate samples, each of which is made up of several different food items. I multiply each food by a specific multiplier to convert it to energy, and then sum the energy ...
1
vote
1answer
62 views

How to control for type-1-error inflation with multiple chi-squared tests?

I have two between-subject factors, each with two levels (so 4 conditions). Furthermore, I have one dependent variable (qualitative), also consisting of two levels. Now I want to make pairwise ...
1
vote
0answers
22 views

Multcomp glht function default

Does anyone know the default glht multiple comparison adjustment used when i specify contracts in linfct? Example: glht(md.fit,linfct=(contr))
0
votes
0answers
16 views

Decision Level Fusion of SVR outputs

I have two sets of features predicting the same outputs. But instead of training everything at once, I would like to train them separately and fuse the decisions. In SVM classification, we can take ...
1
vote
0answers
22 views

statistical comparison of power spectrums over all frequencies

So I have N=147 control time-series signals and N=134 treatment (THC) signals. I want to compare if there are any significant differences in frequency power in the two groups. The top part of the ...
0
votes
0answers
18 views

Comparing density plots and scoring the combination

I have a set of density plots that contain the distribution of stock prices. Each graph has 5 density plots as follows that shows the distribution of monthly returns based on their ratings - ...