Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
4 views

When to correct p-values in multiple comparisons?

I'm afraid that related questions didn't answer mine. We evaluate the performances of >2 classifiers (machine learning). Our Null hypothesis is that performances do not differ. We perform parametric ...
2
votes
2answers
114 views
+50

Comparing performances during a race between 2 groups, which test should I use?

Imagine I organized a race between 2 types of ants. Type A and B. At the end of the race, I have the ranking, which looks something like : ...
3
votes
1answer
14 views

Multiple comparisons in a non-parametric test

I am working with a data set that is non-parametric and has 12 treatments. I performed the Kruskal-Wallis test and got a significant $p$-value, and now I would like to conduct a multiple comparisons ...
0
votes
1answer
41 views

FDR (multiple testing correction) with skewed p-value distribution

I am confused with multiple comparisons adjustments. I have a $p$-values with lot of ones ( due to many scores in foreground are 0) from a fisher-exact test. I get some $p$-values which are ...
2
votes
1answer
54 views

When should I use FDR or Bonferroni in multiple comparisons?

I have the following situations : (1) Very large number of variables - say 100 variables and 200 samples (observations). Although level of each variable is just 2 or 3 or 4 max. I am doing linear ...
1
vote
2answers
35 views

Multiple comparison on subgroups and overall comparison

I am trying to work out is there is any association between occurrence or not of an event in around 500 individuals and their randomised binary grouping (intervention vs placebo). However I also want ...
0
votes
2answers
17 views

What are correlations among between test statistics in multiple comparison procedures

In the book Multiple Comparisons Using R Bretz et al write that their R package multcomp improves power above other multiple comparison procedures by "accounting for the structural correlations ...
1
vote
0answers
17 views

Location test of the ratios of random variables

I've been struggling with this problem for a while now, with not much progress. Any suggestions would be helpful! Suppose I have $X_1, .., X_5, Y_1, ..., Y_5, X'_1, ..., X'_5, Y'_1, ..., Y'_5$, i.e., ...
0
votes
0answers
15 views

Multiple testing correction when confirming a correlation with an extended dataset

Consider a given dataset on which I performed a set of independent tests on different (non-independent) variables, and obtained several p-values. I estimated the number of independent tests as being ...
0
votes
0answers
18 views

Which multiple comparisons correction should I use?

I'm studying structure-function relationships in neurons and mainly focusing on correlations between dendritic length and excitability. My question is: When searching for correlations, which ...
1
vote
2answers
58 views

Multiple testing: contingency table or logistic regression approach for survey?

I have a survey asking a person's gender (M/F), geographic region (West, East, Midwest, South) and a few other demographic variables along with two dozen questions in which they can disagree / ...
1
vote
0answers
10 views

Correcting for multiple comparisons using simulated distribution

To check if unilateral pairs (defined below*) are coordinated (i.e. move together) in a flock of N coordinated individuals, we generate hypothetical (null) distribution of a certain focal parameter of ...
2
votes
0answers
34 views

MANOVA vs repeated measues ANOVA example

I have the following situation (4 independent treatment groups, each group is tested for 3 dependent variables) and was wondering what would be the best to do: What is the best test to do? I have ...
5
votes
1answer
152 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
1
vote
0answers
72 views

Multiple Hypothesis testing correction on non-independent data

Consider a vector $X_i$ with $i \in [1,N]$. Let us say that you divide the vector into two $X_{i}'$ with $i \in [1,j]$ and $X_i''$ with $i \in [j+1, N]$ for several $j$; then for each pair of ...
0
votes
0answers
16 views

ANOVA, unbalanced designs, missing data, and multiple comparisons

I am having several problems with my dataset and how best to analyze it. I have measured a series of plant phenology characteristics (with a seperate model for each one - I do not want to combine them ...
0
votes
1answer
22 views

How to summarize multiple adjusted pvalue into a single measure?

Suppose I have a test with multiple hypotheses. For example, if I have X as a gene set (a set of symbols), and A, B, C pathways (sets of symbols), I am testing if A is enriched in X, B in X, C in X ...
0
votes
0answers
19 views

Levene's test for multiple comparisions

by doing ANOVA of biological data from four species I've find that my data have unequal variance between the species. Now I would like to determine which species are different in this character. Is it ...
3
votes
0answers
61 views

Multiple comparisons with interactions of mixed lmer model and how to report them

Assume I test a number of patients repeatedly over time to see how a certain treatment changes their skin conductance in response to a certain colour (cond) after 2 months, 4 months, ... etc. I test ...
0
votes
0answers
40 views

Multiple hypothesis testing or simply p-value sum?

For a given dataset I test simultaneously $m$ different hypothesis, obtaining a p-value for each. Each of these hypothesis correspond to test if the values $X_i$, a measured quantity from the dataser, ...
0
votes
0answers
13 views

Multivariate multisample equivalence test

I have multiple observations from n groups\treatments. How to test the equivalence of n groups using all the variables? ...
1
vote
0answers
14 views

Why does max-t methods use only the maximum of generated t-values?

I am wondering why max-t methods use the maximum of the t-values to generate the distribution. Does this not make it overly conservative compared to other methods, since the critical value will be ...
0
votes
1answer
29 views

Alternative to Friedman test for single treatments (no repetitions)

I'm looking for a non-parametric (!) alternative to the Friedman test. I have > 2 subjects and measured one quantity per subject. I want to know whether there is a significant difference between any ...
1
vote
0answers
18 views

Corrected multiple comparison: is the preceeding ANOVA necessary at all? [duplicate]

To compare multiple groups, one usually runs ANOVA. If ANOVA rejects the null hypothesis, one goes through the multiple comparisons. The multiple comparisons are necessary to establish which group is ...
0
votes
0answers
24 views

Identifying formulas from MANOVA in a research paper

I need to identify the formulas for computing the group differences with Bonferroni corrections and Cohen's d effect size in the following paper (chart on the top of page 6): Research Paper What ...
0
votes
1answer
20 views

Correction for multiple comparisons with multiple response variables

I'm trying to determine what constitutes a family of hypotheses when measuring multiple response variables. In particular I have an A/B type test where I'm comparing several marketing campaigns ...
4
votes
1answer
135 views

False discovery rate of multiple regressions models

I have 10 models analysing pupal developmental rates (DR) and developmental times (DT) depending on a number of factors: ...
3
votes
1answer
87 views

Machine learning classifiers big-O or complexity

To evaluate the performance a new classifier algorithm, I'm trying to compare the accuracy and the complexity (big-O in training and classifying). From Machine Learning: a review I get a complete ...
0
votes
0answers
14 views

Advice on how to analyze these data

In my experiment, I have 312 separate samples, each of which is made up of several different food items. I multiply each food by a specific multiplier to convert it to energy, and then sum the energy ...
1
vote
1answer
54 views

How to control for type-1-error inflation with multiple chi-squared tests?

I have two between-subject factors, each with two levels (so 4 conditions). Furthermore, I have one dependent variable (qualitative), also consisting of two levels. Now I want to make pairwise ...
1
vote
0answers
19 views

Multcomp glht function default

Does anyone know the default glht multiple comparison adjustment used when i specify contracts in linfct? Example: glht(md.fit,linfct=(contr))
0
votes
0answers
15 views

Decision Level Fusion of SVR outputs

I have two sets of features predicting the same outputs. But instead of training everything at once, I would like to train them separately and fuse the decisions. In SVM classification, we can take ...
1
vote
0answers
20 views

statistical comparison of power spectrums over all frequencies

So I have N=147 control time-series signals and N=134 treatment (THC) signals. I want to compare if there are any significant differences in frequency power in the two groups. The top part of the ...
0
votes
0answers
16 views

Comparing density plots and scoring the combination

I have a set of density plots that contain the distribution of stock prices. Each graph has 5 density plots as follows that shows the distribution of monthly returns based on their ratings - ...
0
votes
1answer
26 views

How to find correlation among dependent variables?

If I want to find how strongly a dependent variable is related to another dependent variables in a study, do I make use of multiple regression? The reason I am asking is because the book mentions ...
1
vote
1answer
41 views

Partially orthogonal planned comparisons

I am comparing 8 means and want to set up a planned comparisons, rather than having my Bonferroni adjustment become overly-conservative in a post-hoc. For my groups I need to make a total of 16 ...
5
votes
2answers
85 views

Help for planning a neuroscience experiment

I'm new here. I am planning a neuroscience experiment. I will be measuring brain signals from about twenty subjects. I will present the subjects four different kinds of stimuli. After all the data ...
0
votes
1answer
51 views

Permutation tests and multiple testing

I'm using permutation tests, based on random forest importance measures, to perform variables selection in my dataset. Using $99$ permutations, my permutation tests $p$-values are comprised between ...
2
votes
1answer
137 views

Hypothesis testing on Poisson Binomial distribution

Imagine $N$ baskets each filled with a different number of products chosen from a set $P$. It is possible to have repeated products in the same basket. For instance: ...
1
vote
1answer
44 views

Multiple testing problem - can these test results be caused by chance?

I conduct 30 independent significance tests on $p<.05$. I observe that in $X=3$ cases the test is rejected. Can this event be caused by chance due to a multiple testing problem? I understand that ...
0
votes
0answers
11 views

non significance with one-way ANOVA, but significance with multiple comparisons [duplicate]

I'm comparing deviation between three directions (vrt, lng and lat). I applied ...
1
vote
0answers
62 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...
0
votes
1answer
45 views

Grouping predictor factor levels based on response variable

I've read that it's bad to do this, but am looking for details as to why. Suppose we're trying to fit the linear model $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \epsilon_i$ where $Y$ is ...
0
votes
1answer
154 views

Why is the Tukey confidence interval wider than the T-test confidence interval?

I have two confidence intervals for a comparison of two means. The first is a t-test CI and the other is a Tukey CI. The Tukey CI $(2.36, 26.5)$ is wider than the t-test CI $(6.46, 21.33)$. What is ...
0
votes
0answers
42 views

How to test measurement invariance with PLS?

I have two questions about structural equations modeling: To perform comparisons of model estimates for different groups (3 clusters), do I always have to analize measurement invariance first? If ...
0
votes
0answers
15 views

Multiple comparisons with Dunnetts t3 or Dunnetts c

I have a dataset with un-even variance and un-equal class sizes (This was a random sample of land cover classes). After using ANOVA to determine significant differences exist among classes, I ran ...
0
votes
0answers
36 views

Multiple comparison in the unequal variance case

I have found procedures such as Tamhanes T2, Dunnets T3 and the Games & Howell procedure that deal with unequal variances in the one-way model. However, I have a Randomized complete block design, ...
2
votes
1answer
128 views

Does one need to adjust for multiple comparisons when using MANOVA?

There are 10 dependent variables and 2 groups. When I wanted to compare 10 dependent variables between two groups, I performed MANOVA. I wonder if there is a multiple comparison problem in that case. ...
4
votes
1answer
104 views

Does R have post hoc tests robust to unequal sample sizes/population variances?

While reading Discovering Statistics Using R pp. 431-432, Dr. Field says that "There are a variety of tests designed to deal with these situations [multiple comparison procedures with unequal ...
0
votes
0answers
28 views

When are multiple linear regressions independent?

I'm doing many pairwise linear regressions on a set of variables. For example x as the dependent variable vs y, ...