Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
7 views

glht on gamm results

I have a GAMM model that has both an LME and a GAM part. I have used glht (multcomp) to check significant differences through the code: ...
0
votes
0answers
7 views

Fmax permutation for multiple comparisons [duplicate]

In an EEG experiment we presented stimuli from four conditions. This was a within-subject design. Dependent variable was the amplitude of brain activity. For each pre-defined regions of electrodes on ...
1
vote
1answer
24 views

simulating a multiple comparisons problem using R and bonferroni correction

I am actually working on an exercise for my students dealing with multiple comparisons. However, I was having trouble trying to simulate some data in R to demonstrate the problem with multiple ...
0
votes
0answers
45 views

Edited: Glmer model won't run all factor levels due to multicolinearity - can you have pseudo-multicolinearity?

I have a repeated measures design in which I have tested 112 individuals each with three different 20 second playback stimuli. I used a random block design where an individual was assigned to 1 of 6 ...
0
votes
1answer
46 views

Can assumptions of a statistical test be tested post-hoc?

Suppose I ran 100 1-factor ANOVAs. I find some significant results. Now, since ANOVA has some assumptions, I should test if they hold. So, can I run for example Levene's tests (to check equity of ...
0
votes
1answer
18 views

IBM SPSS QUESTION: Measure the perception of the respondents

My problem to solve with IBM SPSS is: I have got answers to a question about X topic that contains 15 options of multiple response and I want to characterize the perception of the correspondents ...
0
votes
1answer
14 views

How to compare multiple frequency distributions with some zeroes?

I have a data frame with frequencies for the teacher profession descriptors as it is perceived by 1st year, 2nd year and MA students. Some of this frequencies are zeroes. What statistical method can I ...
0
votes
1answer
52 views

Is it wise to use permutation test with many electrodes and many conditions?

I have data from an EEG (electrophysiological data) experiment where we have more than 10 electrodes recording brain activity (dependent variable) simultaneously. The design is a within subject design ...
0
votes
1answer
41 views

Post-tests to consider in mixed models

I have a mixed model of the form y~condition + Replicate + Error(Replicate) but at the moment, and am running into difficulties in performing a post-hoc test. I ...
0
votes
0answers
20 views

Comparison of completion time for CAPTCHAs between human users and computers

CAPTCHAs are used to distinguish legitimate human users from computerized programs that are trying to abuse online services. There is a way to bypass the security by relaying CAPTCHA challenges to ...
0
votes
0answers
14 views

impact of RNGs on EAs

Evolutionary algorithms (EAs) that simulate the effects of "mutation, reproduction, selection, replacement, etc." often incorporate the use of Random Number Generators (RNGs). Different EAs were ...
2
votes
1answer
27 views

Multiple comparisons with many groups

I would like to determine if using multiple comparisons test would be appropriate for my data. I used the Kruskal-Wallis test to determine if there were differences in mean inhibition among $17$ ...
1
vote
1answer
33 views

Bonferonni correction on multiple Kruskal-Wallis tests

My data consists of 3 groups, and 6 variables. The data are not normal so I want to use non-parametric tests to uncover significance. The first thing I wanted to get your opinion on is, is it ...
0
votes
0answers
22 views

Compare to sets of correlations coefficients

I have two matrices, A and B, of the same size. Each row of A corresponds to a row of B. Theory predicts that some rows of A will negatively correlate with corresponding rows of B. I am interesting ...
1
vote
1answer
11 views

Appropriate post-hoc test for many comparisons following 2-way ANOVA

A reviewer has suggested using either the Hochberg step up or Bonferroni step down analysis or similar tests to better control for Type I error by controlling the alpha value for the number of ...
1
vote
0answers
12 views

Testing sequentially nested models

Assume I have a simple model with (at least) four parameters $\beta_1, \beta_2, \beta_3, \beta_4$. If I would want to test $H_o: \beta_1 = \beta_2 \& \beta_3 = \beta_4$ by using the likelihood ...
1
vote
1answer
32 views

Choosing discrimination threshold from ROC curve, is there a multiple testing issue?

When choosing a discrimination threshold from a ROC curve (as nicely explained on mathematical thinking) it strikes me that effectively a hypothesis test has been conducted for each point that is ...
1
vote
0answers
22 views

Multiple testing with one-factor analysis of variance

I have groups A (N=60), B (N=30), and C (N=30), and 100 variables. Using one-factor analysis of variance (F-test), I would like test whether the three samples differ for each variable. So, if I first ...
2
votes
1answer
43 views

When correcting for multiple comparisons, do you correct all p-values or only significant ones?

This feels like a really basic question, but I haven't found a definitive answer. When correcting p-values for multiple comparisons, are you supposed to only run the analysis/correction on the ...
1
vote
1answer
44 views

Statistical test for multiple comparison of groups of animals with Yes or No outcome

In my experiment (pre-clinical vaccine testing) I want to know what kind of statistical test to be used to compare between 9 groups of animals (72 animals randomly divided into 9 groups). Each group ...
3
votes
0answers
50 views

Is Uplift Modeling a solution to Multiple Comparisons problem?

If you want to identify a particular user segment for whom an experiment produced some lift or incremental effect over the control treatment, wouldn’t it be more direct to do uplift modeling? Uplift ...
0
votes
1answer
21 views

How many hypotheses are needed when researching one category with multiple likert-scale question

I am currently researching two groups on their willingness with 5 questions on a 7-point Likert-scale. My hypotheses are: $H_0$: There is no difference in willingness between the two groups. $H_1$: ...
0
votes
0answers
31 views

Is there a non-parametric repeated measures test for replicated block data?

Following on from an earlier question, I've got another question about comparing machine learning classifiers on various datasets. From the response to that question, and also to this paper I ...
0
votes
0answers
22 views

Significance of multivariate models and correction for multiple comparisons

I have performed a multivariate binary classification using a number of features (or variables), I will call them features from sets (A), (B) and (C). I have calculated the P value of this ...
3
votes
1answer
53 views

p-value correction for multiple t-tests?

My dataset consists of $n$ genes, each of them described by a vector of expression values, $5$ for "healthy" individuals, and $5$ for "unhealthy" individuals. I am going to run $n$ t-tests (one for ...
1
vote
1answer
42 views

If I have to run 500 statistical tests, should I consider each case separately or find the “fittest” one technique?

Suppose I have to run 500 tests. For example, compare two groups with respect to 500 variables. If for a given variable the two samples are normally distributed, I could do a Student's independent two ...
0
votes
1answer
45 views

Adjusting for multiple Kruskal-Wallis tests

I've run 3 different machine learning algorithms on 10 different datasets, generating an accuracy on each one. My hypothesis is that two of the algorithms are consistently better than the third. I've ...
0
votes
0answers
20 views

Pairwise Wilcoxon test gives wrong results

I'm trying to perform a pairwise Wilcoxon test in R. When using Bonferroni p-adjustment I got only 1's and 0's. When changing p-adj to 'none', I get this: ...
0
votes
0answers
27 views

What is the relationship between degrees of freedom and number of comparisons in Nemenyi's post hoc test?

I have some data consisting of non-normally distributed samples drawn from one of several different groups. A Kruskal-Wallis 1-way ANOVA rejects the null hypothesis that the medians of the groups are ...
2
votes
0answers
45 views

Running sequential tests as data is collected - practical advice for researchers

I've observed that many experimental labs in my field don't do a power analysis before they start collecting data. In practice, it seems that many researchers collect data and periodically run ...
0
votes
1answer
42 views

confidence intervals best through direct calculation or R's confint funtion

I am interested in pairwise comparison of means, through the calcul of confidence intervals (CI) of the difference between two given sample's means. However, CI calculated directly from each sample, ...
3
votes
3answers
238 views

Do I need to correct for multiple hypotheses? p-values all go to one

I have two groups, C and A, where C is control. In both groups there are about 3,000 proteins. For each protein we acquire six measurements in both C and A (these are mass spectrometry intensities by ...
0
votes
2answers
65 views

multiple significance testing of large data

I am doing analysis using multiple significance testing between about 20 parameters of data with >8000 rows. Since the number is reasonably large, even minor differences are coming with highly ...
0
votes
1answer
17 views

Correction of mutliple paired samples t-test

In one group of subjects I have four parameters (for e.g. A, B, C, D) measured in two conditions. I want to know whether these 4 parameters differ between the conditions by doing paired samples ...
1
vote
0answers
41 views

Tukey HSD, insignificant p-value when the confidence interval does not contain 0?

I am using TukeyHSD to do multiple comparisons. I am seeing strange results for some of the pairs: the adjusted p-value is greater than the specified significance level but the corresponding ...
0
votes
0answers
27 views

Multiple comparisons of proportions using complex survey data using SAS

I am interested in doing multiple comparisons of proportions using complex survey data in SAS. I can get estimates using proc surveyfreq or proc crosstab (using SAS-callable SUDAAN), but am not aware ...
1
vote
0answers
72 views

Comparison and testing multinomial coefficients

Straight to the problem I want to test the hypothesis that two coefficients of multinomial regression model are equal between them. For example, considering a model where Y is 3 level categorical ...
0
votes
1answer
181 views

How do I interpret a variance ratio test in Stata?

I need help with interpreting output from a Variance Ratio Test in Stata. $f = 1.2988$ (my variance) $\text{Degrees of freedom} = 249, 493$ I have Ho: ratio = 1 as my null Ha: ratio != 1 Am I ...
1
vote
1answer
36 views

Adjust the p-values or not?

This started as a simple discussion where everyone thought they new the answer, and ended up with arguments and dozens of paper quotations. Thoughts appreciated. We have 10 samples: Control, A, B, C, ...
1
vote
1answer
126 views

Multiple testing in correlation analysis over time periods

I have one variable measured once per time interval (say, once per year), and another variable measured periodically (say, once per day). The periodic measurements are autocorrelated. I am interest ...
0
votes
0answers
22 views

Comparing the difference in proportions for 3 different groups

I conducted an experiment and just need some general guidance on where to go from here. I know I have to do some kind of binomial test on my data. Suppose I have the following data: ...
0
votes
0answers
27 views

Multiple Comparison of Standard Deviations

I have a question about comparing multiple standard deviations: To start with, let's say I have calculated two standard deviations (s1, s2) from a series of random samples. If I want to determine ...
1
vote
0answers
25 views

get p value based on observed F and critical F

When conducting multiple comparisons, I can calculate a critical value via whatever method is appropriate for the situation. If the adjustment method I need is a common one such as Bonferroni, I can ...
0
votes
0answers
25 views

Do we need to correct p-values when using multiple models?

I am currently studying epidemiology. I am writing a paper that is examining 11 disease states in one species. I am using binomial logistic regression to examine 2 possible predictors (age and ...
1
vote
0answers
54 views

Estimating and comparing pre-post treatment effects in multiple pairs of treatment-vs-control groups

I have a dataset from an experiment where the goal was to study in which plant ecotype a treatment with a certain bacteria had the greatest effect. Hence the data consists of a number of control and ...
0
votes
0answers
8 views

Null distribution of the maximum and FDR control

Suppose we have a series of test statistics $T_{1},...,T_{m}$ each corresponding to a null hypothesis $H_{0}^{j}: \theta_{j} = 0$. Let $H_{0}: \theta_{1} = \ldots = \theta_{m} = 0$ denote the ...
1
vote
2answers
118 views

Any other non-parametric alternative to Kruskal-Wallis?

It looks like Kruskal-Wallis is the standard nonparametric test for more than two groups. The problem is that it does not tell which groups are different, except that whether there exists significant ...
0
votes
0answers
34 views

Rank sums in the Kruskal-Wallis output

I use Stata to perform a Kruskal-Wallis test on four groups of data (not normally distributed), and the results are shown below: ...
3
votes
0answers
130 views

How to best approach ANOVA with multiple response variables that might be correlated and Bonferroni Correction

I need some help verifying a simple analysis. I have completed a simple two factor experiment of more or less equal sample sizes. Let's assume for the case of this question that the sample sizes are ...
0
votes
1answer
119 views

Which post-hoc is more valid for multiple comparison of an unbalanced lmer-model: lsm or mcp?

After doing a model comparison with my mixed lmer model, I have a model with three main effects, no interaction, say ...