Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
15 views

Confusion with FDR and multiple testing

I have read this great paper by David Colquhoun. In essence, he explains why FDR can be as high as 30% even though we control for type I error with α=0.05. However I am still confused as to what ...
0
votes
0answers
23 views

Vector Statistical Significance [on hold]

PS: If possible in R or math I am looking for the best way to do significance testing, but given multiple variables. Looking at my example below, there are 2 groups, and 4 variables. Each row is a ...
0
votes
0answers
23 views

Optimal model/statistical test for my design?

I have a design with 1 between-subject factor and 2 within-subject factors as independent variables and 5 dependent variables (longitudinal accelartion, lateral acc., response time, first conscious ...
0
votes
0answers
18 views

Pairwise comparisons with covariates

I have a simple ANCOVA model such that outcome ~ sex + group with outcome being continuous, ...
0
votes
0answers
19 views

Kruskal-Wallis post hoc comparisons different results in different software used

I'm having some trouble analyzing my data because when I use StatSoft Statistica and run a Kruskal-Wallis, I get one set of values for multiple comparisons, but when the same analysis is run in XLSTAT ...
0
votes
0answers
14 views

Dunnett for comparing multiple means to a specific value

In text-book statistics, one can use the one-sample t-test for comparing a population mean to a specific value. What would be the correct way to conduct a Dunnett's procedure of comparing multiple ...
3
votes
0answers
13 views

Multiple comparisons of parameters from non-linear regression

Hei, I want to compare parameters a_i and b_i estimated by nonlinear regression (y=a_i*x/(b_i+x)) for different data-sets (let's say 8 different data sets). I have calculated the non-linear ...
1
vote
0answers
23 views

Multiple t tests or an ANOVA?

I have low anxiety and high anxiety participants (21 in high anxiety and 20 In low). I measured anxiety and then categorised Pts according to this. They completed two memory tasks. In one memory ...
0
votes
0answers
33 views

Significant Friedman's test vs. non-significant multiple comparisons

I am currently running a randomized, placebo-controlled, cross-over design for a new type of intervention. This intervention is compared to a placebo and another more conventional type of treatment. ...
0
votes
0answers
36 views

compare muliptle probabilities

I have built three decision tree model to predict the response for three offers (One model per offer). I want to find the best offer for each customer based on the predicted probabilities. All offers ...
4
votes
2answers
139 views

Using multiple comparisons corrections in Fisher p-value framework

I am wary of stepping into the civil war between Fisher vs Neyman-Pearson interpretations of $p$-value (which has been well elucidated here and here), but I've been pondering a question that I keep ...
0
votes
0answers
45 views

P-values adjustment with known results of some hypotheses

I ran an experiment - within-subjects design; 2 scenarios (congruent and incongruent one). In each scenario I measured overall task completion time, geo. mean of task completion time and accuracy. ...
0
votes
0answers
18 views

Multiple testing confusion, what is significant?

I'm learning multiple hypothesis testing and this confuses the hell out of me. This is just a simple "are these distribution significantly different?" question. Data: Treatment: 100k random ...
0
votes
0answers
18 views

Which means difference test to use?

Currently, my data is count data and I am analyzing it in Proc Glimmix in SAS. My data focuses on how many times birds from two treatments performed stand, sit, forage, ect.... So it basically looks ...
0
votes
1answer
39 views

How to calculate critical values for Dunnett procedure given alpha, df1 and df2

I am looking for an R function that can calculate critical values for a Dunnett multiple comparison procedure for any family-wise error rate and degrees of freedoms. In short, I need similar ...
1
vote
0answers
30 views

Independent t test significant but post-hoc planned analysis not

I was analysing two sets of data. In most cases for the study post hoc planned comparisons identified significance (<0.05), however for one sample, this was not achieved. However an independent t ...
1
vote
0answers
63 views

How much confidence can we have in statistically-based ranking for “safe” international airlines? [closed]

Some cites for such ratings exist, but they are not meant to be predictive. For example AirlineRatings.com awards stars based on questions like Is the airline on the European Union (EU) Blacklist? ...
1
vote
1answer
13 views

How to Compare more than two Averages of Proportions or Means of Proportions

My experiment deals with $60$ subjects, one third of them belong to type A subjects, $i=1..20$, another third to type B, $i=21…40$ and finally type C, $i=41..60$. Each of these $60$ subjects from the ...
2
votes
2answers
74 views

Is it appropriate to do an ANOVA on a feature selected via inspecting PCA results?

I've been given a dataset consisting of 8 dimensional feature vectors for 4 classes of objects. I was asked to find the features that best distinguish the classes, and write up a short report. My ...
0
votes
0answers
15 views

Multiple comparisons with genetic * environment interactions

I am analysing the next models with logistic or linear regression: Y = G*E + confounders, where G is a genetic variable (SNP) and E is an environment variable. The problem is the next: I have 13 SNPs ...
7
votes
1answer
99 views

Plain language meaning of “dependent” and “independent” tests in the multiple comparisons literature?

In both the family-wise error rate (FWER) and false discovery rate (FDR) literature, particular methods of controlling FWER or FDR are said to be appropriate to dependent or independent tests. For ...
2
votes
2answers
65 views

Creating groups using two continuous variables without using median-splitting?

I have two continuous variables, one of individuals' retrospective childhood anxiety and another regarding their current level of anxiety. Research has demonstrated that during a snapshot of ...
0
votes
1answer
40 views

Multiple T Tests

On one of my questionnaires, which measure learning behaviours, after PCA, I have 4 subgroups, Efficacy/Perseverance/Effort/Achievement - I have run a T-Test and I have 4 sets of data comparing my ...
1
vote
0answers
35 views

Running 2x2x2 ANCOVA in SPSS

I need to conduct an ANCOVA on my data. While I've taken a number of statistics courses, I have never come across ANCOVAs and I want to get some feedback on the correct way to conduct this analysis. ...
0
votes
0answers
17 views

glht on gamm results

I have a GAMM model that has both an LME and a GAM part. I have used glht (multcomp) to check significant differences through the code: ...
0
votes
0answers
9 views

Fmax permutation for multiple comparisons [duplicate]

In an EEG experiment we presented stimuli from four conditions. This was a within-subject design. Dependent variable was the amplitude of brain activity. For each pre-defined regions of electrodes on ...
1
vote
1answer
35 views

simulating a multiple comparisons problem using R and bonferroni correction

I am actually working on an exercise for my students dealing with multiple comparisons. However, I was having trouble trying to simulate some data in R to demonstrate the problem with multiple ...
0
votes
0answers
64 views

Edited: Glmer model won't run all factor levels due to multicolinearity - can you have pseudo-multicolinearity?

I have a repeated measures design in which I have tested 112 individuals each with three different 20 second playback stimuli. I used a random block design where an individual was assigned to 1 of 6 ...
0
votes
1answer
52 views

Can assumptions of a statistical test be tested post-hoc?

Suppose I ran 100 1-factor ANOVAs. I find some significant results. Now, since ANOVA has some assumptions, I should test if they hold. So, can I run for example Levene's tests (to check equity of ...
0
votes
1answer
37 views

IBM SPSS QUESTION: Measure the perception of the respondents

My problem to solve with IBM SPSS is: I have got answers to a question about X topic that contains 15 options of multiple response and I want to characterize the perception of the correspondents ...
0
votes
1answer
22 views

How to compare multiple frequency distributions with some zeroes?

I have a data frame with frequencies for the teacher profession descriptors as it is perceived by 1st year, 2nd year and MA students. Some of this frequencies are zeroes. What statistical method can I ...
0
votes
1answer
100 views

Is it wise to use permutation test with many electrodes and many conditions?

I have data from an EEG (electrophysiological data) experiment where we have more than 10 electrodes recording brain activity (dependent variable) simultaneously. The design is a within subject design ...
0
votes
1answer
69 views

Post-tests to consider in mixed models

I have a mixed model of the form y~condition + Replicate + Error(Replicate) but at the moment, and am running into difficulties in performing a post-hoc test. I ...
0
votes
0answers
21 views

Comparison of completion time for CAPTCHAs between human users and computers

CAPTCHAs are used to distinguish legitimate human users from computerized programs that are trying to abuse online services. There is a way to bypass the security by relaying CAPTCHA challenges to ...
0
votes
0answers
14 views

impact of RNGs on EAs

Evolutionary algorithms (EAs) that simulate the effects of "mutation, reproduction, selection, replacement, etc." often incorporate the use of Random Number Generators (RNGs). Different EAs were ...
2
votes
1answer
53 views

Multiple comparisons with many groups

I would like to determine if using multiple comparisons test would be appropriate for my data. I used the Kruskal-Wallis test to determine if there were differences in mean inhibition among $17$ ...
1
vote
1answer
100 views

Bonferroni correction on multiple Kruskal-Wallis tests

My data consists of 3 groups, and 6 variables. The data are not normal so I want to use non-parametric tests to uncover significance. The first thing I wanted to get your opinion on is, is it ...
0
votes
0answers
30 views

Compare to sets of correlations coefficients

I have two matrices, A and B, of the same size. Each row of A corresponds to a row of B. Theory predicts that some rows of A will negatively correlate with corresponding rows of B. I am interesting ...
1
vote
1answer
16 views

Appropriate post-hoc test for many comparisons following 2-way ANOVA

A reviewer has suggested using either the Hochberg step up or Bonferroni step down analysis or similar tests to better control for Type I error by controlling the alpha value for the number of ...
1
vote
0answers
14 views

Testing sequentially nested models

Assume I have a simple model with (at least) four parameters $\beta_1, \beta_2, \beta_3, \beta_4$. If I would want to test $H_o: \beta_1 = \beta_2 \& \beta_3 = \beta_4$ by using the likelihood ...
1
vote
1answer
48 views

Choosing discrimination threshold from ROC curve, is there a multiple testing issue?

When choosing a discrimination threshold from a ROC curve (as nicely explained on mathematical thinking) it strikes me that effectively a hypothesis test has been conducted for each point that is ...
1
vote
0answers
23 views

Multiple testing with one-factor analysis of variance

I have groups A (N=60), B (N=30), and C (N=30), and 100 variables. Using one-factor analysis of variance (F-test), I would like test whether the three samples differ for each variable. So, if I first ...
3
votes
1answer
53 views

When correcting for multiple comparisons, do you correct all p-values or only significant ones?

This feels like a really basic question, but I haven't found a definitive answer. When correcting p-values for multiple comparisons, are you supposed to only run the analysis/correction on the ...
1
vote
1answer
87 views

Statistical test for multiple comparison of groups of animals with Yes or No outcome

In my experiment (pre-clinical vaccine testing) I want to know what kind of statistical test to be used to compare between 9 groups of animals (72 animals randomly divided into 9 groups). Each group ...
3
votes
0answers
64 views

Is Uplift Modeling a solution to Multiple Comparisons problem?

If you want to identify a particular user segment for whom an experiment produced some lift or incremental effect over the control treatment, wouldn’t it be more direct to do uplift modeling? Uplift ...
0
votes
1answer
25 views

How many hypotheses are needed when researching one category with multiple likert-scale question

I am currently researching two groups on their willingness with 5 questions on a 7-point Likert-scale. My hypotheses are: $H_0$: There is no difference in willingness between the two groups. $H_1$: ...
0
votes
0answers
65 views

Is there a non-parametric repeated measures test for replicated block data?

Following on from an earlier question, I've got another question about comparing machine learning classifiers on various datasets. From the response to that question, and also to this paper I ...
3
votes
1answer
69 views

p-value correction for multiple t-tests?

My dataset consists of $n$ genes, each of them described by a vector of expression values, $5$ for "healthy" individuals, and $5$ for "unhealthy" individuals. I am going to run $n$ t-tests (one for ...
1
vote
1answer
43 views

If I have to run 500 statistical tests, should I consider each case separately or find the “fittest” one technique?

Suppose I have to run 500 tests. For example, compare two groups with respect to 500 variables. If for a given variable the two samples are normally distributed, I could do a Student's independent two ...
0
votes
1answer
68 views

Adjusting for multiple Kruskal-Wallis tests

I've run 3 different machine learning algorithms on 10 different datasets, generating an accuracy on each one. My hypothesis is that two of the algorithms are consistently better than the third. I've ...