Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
7 views

Significant Friedman's test vs. non-significant multiple comparisons

I am currently running a randomized, placebo-controlled, cross-over design for a new type of intervention. This intervention is compared to a placebo and another more conventional type of treatment. ...
0
votes
0answers
33 views

compare muliptle probabilities

I have built three decision tree model to predict the response for three offers (One model per offer). I want to find the best offer for each customer based on the predicted probabilities. All offers ...
4
votes
2answers
101 views

Using multiple comparisons corrections in Fisher p-value framework

I am wary of stepping into the civil war between Fisher vs Neyman-Pearson interpretations of $p$-value (which has been well elucidated here and here), but I've been pondering a question that I keep ...
0
votes
0answers
44 views

P-values adjustment with known results of some hypotheses

I ran an experiment - within-subjects design; 2 scenarios (congruent and incongruent one). In each scenario I measured overall task completion time, geo. mean of task completion time and accuracy. ...
0
votes
0answers
14 views

Multiple testing confusion, what is significant?

I'm learning multiple hypothesis testing and this confuses the hell out of me. This is just a simple "are these distribution significantly different?" question. Data: Treatment: 100k random ...
0
votes
0answers
16 views

Which means difference test to use?

Currently, my data is count data and I am analyzing it in Proc Glimmix in SAS. My data focuses on how many times birds from two treatments performed stand, sit, forage, ect.... So it basically looks ...
0
votes
1answer
27 views

How to calculate critical values for Dunnett procedure given alpha, df1 and df2

I am looking for an R function that can calculate critical values for a Dunnett multiple comparison procedure for any family-wise error rate and degrees of freedoms. In short, I need similar ...
1
vote
0answers
19 views

Independent t test significant but post-hoc planned analysis not

I was analysing two sets of data. In most cases for the study post hoc planned comparisons identified significance (<0.05), however for one sample, this was not achieved. However an independent t ...
1
vote
0answers
60 views

How much confidence can we have in statistically-based ranking for “safe” international airlines? [closed]

Some cites for such ratings exist, but they are not meant to be predictive. For example AirlineRatings.com awards stars based on questions like Is the airline on the European Union (EU) Blacklist? ...
1
vote
1answer
11 views

How to Compare more than two Averages of Proportions or Means of Proportions

My experiment deals with $60$ subjects, one third of them belong to type A subjects, $i=1..20$, another third to type B, $i=21…40$ and finally type C, $i=41..60$. Each of these $60$ subjects from the ...
2
votes
2answers
64 views

Is it appropriate to do an ANOVA on a feature selected via inspecting PCA results?

I've been given a dataset consisting of 8 dimensional feature vectors for 4 classes of objects. I was asked to find the features that best distinguish the classes, and write up a short report. My ...
0
votes
0answers
13 views

Multiple comparisons with genetic * environment interactions

I am analysing the next models with logistic or linear regression: Y = G*E + confounders, where G is a genetic variable (SNP) and E is an environment variable. The problem is the next: I have 13 SNPs ...
7
votes
1answer
94 views

Plain language meaning of “dependent” and “independent” tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...
2
votes
2answers
51 views

Creating groups using two continuous variables without using median-splitting?

I have two continuous variables, one of individuals' retrospective childhood anxiety and another regarding their current level of anxiety. Research has demonstrated that during a snapshot of ...
0
votes
1answer
36 views

Multiple T Tests

On one of my questionnaires, which measure learning behaviours, after PCA, I have 4 subgroups, Efficacy/Perseverance/Effort/Achievement - I have run a T-Test and I have 4 sets of data comparing my ...
1
vote
0answers
22 views

Running 2x2x2 ANCOVA in SPSS

I need to conduct an ANCOVA on my data. While I've taken a number of statistics courses, I have never come across ANCOVAs and I want to get some feedback on the correct way to conduct this analysis. ...
0
votes
0answers
11 views

glht on gamm results

I have a GAMM model that has both an LME and a GAM part. I have used glht (multcomp) to check significant differences through the code: ...
0
votes
0answers
9 views

Fmax permutation for multiple comparisons [duplicate]

In an EEG experiment we presented stimuli from four conditions. This was a within-subject design. Dependent variable was the amplitude of brain activity. For each pre-defined regions of electrodes on ...
1
vote
1answer
31 views

simulating a multiple comparisons problem using R and bonferroni correction

I am actually working on an exercise for my students dealing with multiple comparisons. However, I was having trouble trying to simulate some data in R to demonstrate the problem with multiple ...
0
votes
0answers
61 views

Edited: Glmer model won't run all factor levels due to multicolinearity - can you have pseudo-multicolinearity?

I have a repeated measures design in which I have tested 112 individuals each with three different 20 second playback stimuli. I used a random block design where an individual was assigned to 1 of 6 ...
0
votes
1answer
51 views

Can assumptions of a statistical test be tested post-hoc?

Suppose I ran 100 1-factor ANOVAs. I find some significant results. Now, since ANOVA has some assumptions, I should test if they hold. So, can I run for example Levene's tests (to check equity of ...
0
votes
1answer
33 views

IBM SPSS QUESTION: Measure the perception of the respondents

My problem to solve with IBM SPSS is: I have got answers to a question about X topic that contains 15 options of multiple response and I want to characterize the perception of the correspondents ...
0
votes
1answer
17 views

How to compare multiple frequency distributions with some zeroes?

I have a data frame with frequencies for the teacher profession descriptors as it is perceived by 1st year, 2nd year and MA students. Some of this frequencies are zeroes. What statistical method can I ...
0
votes
1answer
82 views

Is it wise to use permutation test with many electrodes and many conditions?

I have data from an EEG (electrophysiological data) experiment where we have more than 10 electrodes recording brain activity (dependent variable) simultaneously. The design is a within subject design ...
0
votes
1answer
59 views

Post-tests to consider in mixed models

I have a mixed model of the form y~condition + Replicate + Error(Replicate) but at the moment, and am running into difficulties in performing a post-hoc test. I ...
0
votes
0answers
20 views

Comparison of completion time for CAPTCHAs between human users and computers

CAPTCHAs are used to distinguish legitimate human users from computerized programs that are trying to abuse online services. There is a way to bypass the security by relaying CAPTCHA challenges to ...
0
votes
0answers
14 views

impact of RNGs on EAs

Evolutionary algorithms (EAs) that simulate the effects of "mutation, reproduction, selection, replacement, etc." often incorporate the use of Random Number Generators (RNGs). Different EAs were ...
2
votes
1answer
43 views

Multiple comparisons with many groups

I would like to determine if using multiple comparisons test would be appropriate for my data. I used the Kruskal-Wallis test to determine if there were differences in mean inhibition among $17$ ...
1
vote
1answer
61 views

Bonferonni correction on multiple Kruskal-Wallis tests

My data consists of 3 groups, and 6 variables. The data are not normal so I want to use non-parametric tests to uncover significance. The first thing I wanted to get your opinion on is, is it ...
0
votes
0answers
26 views

Compare to sets of correlations coefficients

I have two matrices, A and B, of the same size. Each row of A corresponds to a row of B. Theory predicts that some rows of A will negatively correlate with corresponding rows of B. I am interesting ...
1
vote
1answer
14 views

Appropriate post-hoc test for many comparisons following 2-way ANOVA

A reviewer has suggested using either the Hochberg step up or Bonferroni step down analysis or similar tests to better control for Type I error by controlling the alpha value for the number of ...
1
vote
0answers
14 views

Testing sequentially nested models

Assume I have a simple model with (at least) four parameters $\beta_1, \beta_2, \beta_3, \beta_4$. If I would want to test $H_o: \beta_1 = \beta_2 \& \beta_3 = \beta_4$ by using the likelihood ...
1
vote
1answer
41 views

Choosing discrimination threshold from ROC curve, is there a multiple testing issue?

When choosing a discrimination threshold from a ROC curve (as nicely explained on mathematical thinking) it strikes me that effectively a hypothesis test has been conducted for each point that is ...
1
vote
0answers
23 views

Multiple testing with one-factor analysis of variance

I have groups A (N=60), B (N=30), and C (N=30), and 100 variables. Using one-factor analysis of variance (F-test), I would like test whether the three samples differ for each variable. So, if I first ...
2
votes
1answer
51 views

When correcting for multiple comparisons, do you correct all p-values or only significant ones?

This feels like a really basic question, but I haven't found a definitive answer. When correcting p-values for multiple comparisons, are you supposed to only run the analysis/correction on the ...
1
vote
1answer
69 views

Statistical test for multiple comparison of groups of animals with Yes or No outcome

In my experiment (pre-clinical vaccine testing) I want to know what kind of statistical test to be used to compare between 9 groups of animals (72 animals randomly divided into 9 groups). Each group ...
3
votes
0answers
62 views

Is Uplift Modeling a solution to Multiple Comparisons problem?

If you want to identify a particular user segment for whom an experiment produced some lift or incremental effect over the control treatment, wouldn’t it be more direct to do uplift modeling? Uplift ...
0
votes
1answer
24 views

How many hypotheses are needed when researching one category with multiple likert-scale question

I am currently researching two groups on their willingness with 5 questions on a 7-point Likert-scale. My hypotheses are: $H_0$: There is no difference in willingness between the two groups. $H_1$: ...
0
votes
0answers
48 views

Is there a non-parametric repeated measures test for replicated block data?

Following on from an earlier question, I've got another question about comparing machine learning classifiers on various datasets. From the response to that question, and also to this paper I ...
3
votes
1answer
65 views

p-value correction for multiple t-tests?

My dataset consists of $n$ genes, each of them described by a vector of expression values, $5$ for "healthy" individuals, and $5$ for "unhealthy" individuals. I am going to run $n$ t-tests (one for ...
1
vote
1answer
43 views

If I have to run 500 statistical tests, should I consider each case separately or find the “fittest” one technique?

Suppose I have to run 500 tests. For example, compare two groups with respect to 500 variables. If for a given variable the two samples are normally distributed, I could do a Student's independent two ...
0
votes
1answer
59 views

Adjusting for multiple Kruskal-Wallis tests

I've run 3 different machine learning algorithms on 10 different datasets, generating an accuracy on each one. My hypothesis is that two of the algorithms are consistently better than the third. I've ...
0
votes
0answers
29 views

Pairwise Wilcoxon test gives wrong results

I'm trying to perform a pairwise Wilcoxon test in R. When using Bonferroni p-adjustment I got only 1's and 0's. When changing p-adj to 'none', I get this: ...
0
votes
0answers
30 views

What is the relationship between degrees of freedom and number of comparisons in Nemenyi's post hoc test?

I have some data consisting of non-normally distributed samples drawn from one of several different groups. A Kruskal-Wallis 1-way ANOVA rejects the null hypothesis that the medians of the groups are ...
2
votes
0answers
46 views

Running sequential tests as data is collected - practical advice for researchers

I've observed that many experimental labs in my field don't do a power analysis before they start collecting data. In practice, it seems that many researchers collect data and periodically run ...
0
votes
1answer
47 views

confidence intervals best through direct calculation or R's confint funtion

I am interested in pairwise comparison of means, through the calcul of confidence intervals (CI) of the difference between two given sample's means. However, CI calculated directly from each sample, ...
3
votes
3answers
241 views

Do I need to correct for multiple hypotheses? p-values all go to one

I have two groups, C and A, where C is control. In both groups there are about 3,000 proteins. For each protein we acquire six measurements in both C and A (these are mass spectrometry intensities by ...
0
votes
2answers
75 views

multiple significance testing of large data

I am doing analysis using multiple significance testing between about 20 parameters of data with >8000 rows. Since the number is reasonably large, even minor differences are coming with highly ...
0
votes
1answer
19 views

Correction of mutliple paired samples t-test

In one group of subjects I have four parameters (for e.g. A, B, C, D) measured in two conditions. I want to know whether these 4 parameters differ between the conditions by doing paired samples ...
1
vote
0answers
49 views

Tukey HSD, insignificant p-value when the confidence interval does not contain 0?

I am using TukeyHSD to do multiple comparisons. I am seeing strange results for some of the pairs: the adjusted p-value is greater than the specified significance level but the corresponding ...