# Tagged Questions

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

51 views

### Group sequential Design

Can anyone recommend a paper which provides a nice introduction or summary of group sequential design.
211 views

### Questions about the glht function for planned comparison

I have a nested model with the following effects fixed: treatments random: experiment_date I used lme() to model the data ...
297 views

### How to analyze categorical variables with multiple levels

I have 14 different habitat classes and two activity states (so two variables – habitat and activity). For activity state A, I have a data count of over 300, but with B, I only have around 30 (so ...
325 views

### Fisher overall p-value vs. pairwise comparisons

I am doing comparisons on response rates across three sites. Following are the cell counts ...
140 views

### Hypothesis testing on data with overlapping groups

I frequently run into the following example: Lot's of genes (order of hundreds of thousands), a few hundred samples stratified into multiple groups. And those groups overlap (i.e. healthy/sick + ...
77 views

### How to compare different heuristics over a large number of test instances

I have 3 different heuristics (H1, H2, H3) that are variation of genetic algorithms combined with some other techniques. The problem that I try to solve is The problem to minimize energy costs for ...
171 views

### Different types of error rates when there are multiple comparisons

I have seen 5 different type of error rates. They include: Componentwise Error Rate Experimentwise Error Rate False Discovery Rate Strong Familywise Error Rate Simultaneous Confidence Intervals ...
35 views

### Is this a valid method to control the FWER?

I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
85 views

### Model comparison across multiple correlated responses

I have two multivariate linear regression models (multiple outcomes, i.e., the responses are a matrix), and I'm measuring their performance using $R^2$ in cross-validation, over these individual ...
52 views

### 3 groups of 8 with a different treatment each, over 45 minutes - confused as to stats test needed

I have three experimental groups. Each will provide a 15 minute baseline before an addition of either drug A, drug B or a saline control. Each group has an N of 8 as maximum, but possibly as low as 6 ...
875 views

### Mixed model multiple comparisons for interaction between continuous and categorical predictor

I would like to use lme4 to fit a mixed effects regression and multcomp to compute the pairwise comparisons. I have a complex ...
442 views

### Is it necessary to correct alpha in repeated measures ANOVA before any post-hoc comparisons?

I have a cross-over design where a sample of subjects undergoes 7 different interventions, and within each intervention there are 3 time points (pre-intervention, post-intervention1, ...
813 views

### Multiple comparisons with the glht function in R gives strange results

As a follow-up to this question, I have the following data: ...
197 views

### Permutation test for the significance of several dependent regression coefficients from separate models

I have $i>100$ Cox regression models which differ only in one explanatory variable, $x_i$. They have the same outcome data and the same other explanatory (adjustment) variables. I want to control ...
142 views

### Using z-Score for multiple comparisons

I have data collected from three groups and I would like to calculate the z-score using the grand mean and variance of the combined three groups. Would it be possible to then compare the z-scores of ...
282 views

### Confidence intervals and pairwise comparisons

Let us say I have the following pair wise test data: ...
340 views

### What are the (best) methods for multiple comparisons correction with bootstrap for multiple glm models?

See the related, but old question: Correcting p values for multiple tests where tests are correlated (genetics). Multiple comparison methods based on bootstrap have the advantage of taking account of ...
209 views

### Hierarchical models for multiple comparisons - multiple outcomes context

I've just been (re-)reading Gelman's Why we (usually) don't have to worry about multiple comparisons. In particular the section "Multiple outcomes and other challenges" mentions using a hierarchical ...
191 views

### Strict Bonferroni - looking for p value

I run 18 exploratory regressions and would like to know what the strict Bonferroni p-value would be. Nine regressions are negative binomial regressions with 4 predictor variables, the other 9 are WLS ...
101 views

### Determine which method predicts best given results from a simulation exercise

This question is related to one I have already asked but the answer I got that suggests I should adopt a new tack to address my research question. I repeat the substantive part of the original ...
149 views

### Does it make sense to average q-values?

I have 3 lists of significant hits (p-value < 0.05). I also adjusted for multiple comparison problem and added q-values to each list. These 3 lists (hits) also partially overlap. For the hits that ...
113 views

### Should false discovery be controlled at the data acquisition level, or should this be at the data interpretation level?

Should false discovery be controlled at the data acquisition level, or should this be at the data interpretation level? I have an experiment in which microarrays were used to quantify the expression ...
59 views

### Merging the results of a statistical test in different conditions and calculating p-value

Suppose we have two mouse strains S1 and S2 and we have already tested the effect of two different conditions (say treatment vs. control) on the level of gene expression of any strain, each experiment ...
347 views

### Independence of residuals in a computer-based experiment/simulation?

I conducted a computer-based assessment of different methods of fitting a particular type of model used in the palaeo sciences. I had a large-ish training set and so I randomly (stratified random ...
257 views

### Benjamini-Hochberg dependency assumptions justified?

I have a data set where I test for significant differences between three populations with respect to some 50 different variables. I do this using Kruskal-Wallis tests, on one hand, and by likelihood ...
173 views

### Overall type I error with dependent tests

Imagine we test 2 dependent samples: For example, we first test an overall sample (treatment vs. control) and then a subset of this overall sample (treatment sbg vs. control sbg). In this case, we are ...
160 views

### Survival analysis w/ repeated observations of experimental and control units within periods of differing size & trends over the periods

I have an experiment called Fear Conditioning Acquisition Protocol. We expose mice to subsequent white sounds followed by short electric shocks through the floor. Our aim is to let them learn to ...
68 views

### Regression and multiple comparison methods for data where each available category is ranked

I have collected a data from 200 customers on their brand preference of 5 international fast food chain shops. Each customer was asked to rank all the 5 brands from highest to lowest (5 for the most ...
1k views

### How to calculate power (or sample size) for a multiple comparison experiment?

I collected data on 20 groups (with 30 elements each). A multiple comparison procedure (pairwise t-test with Holm correction) shows that in general there are three sets of groups: the high with 4 ...
144 views

### What is the colloquial name for the “look elsewhere” effect?

The particle physics community tends to report the significance of a discovery by first calculating the p-values over a range of hypotheses (i.e. values of mass for the hypothetical particle), and ...
74 views

### Multiple comparison with genotype data

I am looking at the effect of genotype on a continuous variable. In total 16 comparisons were done involving 9 different SNPs. The number of significant events was 3 out of 16 tests (Mann-Whitney ...
424 views

### What are the assumptions of the Games-Howell multiple comparisons procedure?

What are the assumptions of the Games-Howell multiple comparisons procedure? Please back up any answers with references.
93 views

### Number of tests for rare events and multiple comparisons problem?

I am conducting some statistical tests on a rare event (11 cases out of 171 patient sample). I have two main hypotheses, but in the mean time I also want to conduct some exploratory tests on 6 other ...
486 views

### How to run a two-way ANOVA with a random variable followed by pairwise comparisons?

I have a dataset with which I would like to compare the effect of species and habitat on movement rate, with pairwise comparisons. I would also like to include the effect of individual (as a random ...
245 views

### lmPerm p-values and multiple testing

I've started using lmPerm in order to perform regressions in R. The equation I want to fit has the form: ...
158 views

### Differential expression gene analysis

I have gene expression data in two samples like this: ...
71 views

### Controlling False Discovery Rate in Stages

I have a three dimensional table of size $6\times6\times81$. Each cell of the table is a hypothesis test. Slicing the table on the third dimension produces $81$ sets of hypothesis tests which are ...
111 views

### How to draw a Hasse diagram of a large number of pairwise comparisons?

I have 100 pairwise comparisons where my results show that 33% of the pairwise comparison tests are non-significant. How can I draw draw a Hasse diagram to represent this? Are there any tips to ...
202 views

### Do I need P value Adjustment for 100 pairwise comparisons?

I made 100 comparisons by applying chi-square test (tajima relative rate test) and got 100 p values. Do I need P-value adjustment? If yes then which one correction will be suitable? I have explored ...
2k views

### How can you have a non-significant multiple regression model w/ significant predictors? [duplicate]

Possible Duplicate: Not-significant F but a significant coefficient in multiple linear regression How can a regression be significant yet all predictors be non-significant? Significance of ...
263 views

### On the applicability of Benjamini-Hochberg

If I've understood correctly, the Benjamini-Hochberg (BH) correction is used to correct for the rate of false discoveries (FDR) when testing a collection of $m$ random variables, $\{X_1, \ldots,X_m\}$ ...
122 views

### How to I compare the growth differences between two time based datasets?

(See note**) I am attempting to put together an infograph and am almost done collecting all the data I need. In addition to the base raw data collected, I'd like to add something I'm calling ...
181 views

### What test should I choose if I want to see if two groups are different from each other in many categories?

I am currently analysing data of a big webshop with over 31,000 vendors selling goods, i.e. each vendor is selling items over the webshop. The cool thing is that we know the gender of these vendors ...
190 views

### Considering residuals as new dependent variables after applying a linear mixed effect model to a set of data

I have a dataset including for each subject : 22 DV (linears, range [0-1]. These DV are intercorrelated). These measures are volume of different brain regions. 1 main effect to test (disease status; ...
185 views

### Repeated measurements with missing values

The data is measured in two biological replicates with two technical replicates, each. Each variable might be detected only in one or more replicates, in the others it is considered missing at ...
781 views

### Does the Tukey HSD test correct for multiple comparisons?

After performing an ANOVA and then doing post hoc tests with a Tukey HSD test is it still necessary to correct for multiple pairwise comparisons or is that done already?
162 views

### Pairwise comparisons in accordance with regression fit

I have a unconventional question arising from my current master thesis on regression modeling. Suppose we have fitted a (linear) relationship between a dependent variable $y$ and an independent ...
281 views

### What are the possible multiple comparison methods for paired data?

I have been working with simulations, among them scheduling simulations. Each simulation chooses the number of steps, the arrival rate of new jobs and the duration of each step for each job. Thus, ...