# Tagged Questions

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

143 views

### If I'm calculating FDR or q-value for a collection of linear models, should each term be considered separately?

I sometimes have the situation where I have from several dozen to over 100 linear models to perform hypothesis tests on. They have the same predictor variables, but different response variables. ...
2k views

### How to interpret a post hoc Tukey's test?

I did a one-way ANOVA followed by a Tukey's test to compare the means of different treatments. Let's say the treatments are A, B and C. The table of multiple comparisons tells me there is a ...
189 views

### Correlation analysis and correcting p-values for multiple testing II

I am currently facing problems dealing with lots of individual simple correlations (28 individual DTI connections strengths, individual behavioural scores on 31 time points). That would be 28x31 ...
159 views

### What test is appropriate for this binned data?

I have collected questions from 10th grade students about genetic engineering before and after learning about the subject and classified the questions based on their cognitive level.. Before learning ...
8k views

### How to obtain the results of a Tukey HSD post-hoc test in a table showing grouped pairs?

I would love to perform a TukeyHSD post-hoc test after my two-way Anova with R, obtaining a table containing the sorted pairs grouped by significant difference. (Sorry about the wording, I'm still new ...
151 views

### Multiplicity in sample size calculation for stratified estimation problem

We want to determine the public opinion about a recently administered intervention in a health care network. There's a brief questionnaire of 10 Y/N questions in which we'd like to estimate ...
422 views

### Use of “post-hoc” tests - do these require an ANOVA to be performed first?

I would like to compare multiple means using a method such as an ANOVA. This would have to be followed up with either Scheffe's test or the Games-Howell post-hoc method (my data is non-normal - to ...
506 views

### Following up a three-way interaction found with (mixed) model comparison

I hope this isn't a silly question, I'd like some advice on following up a threeway interaction in a mixed effects model. I've been building my models incrementally, like this: ...
384 views

### p-value adjustment via permutation for multiple correlations

I have the following problem: I have performed 3200 correlations between a certain measure of connectivity and 1 behavioural outcome in a sample of subjects. As the single correlations (represented by ...
313 views

### Kaplan-Meier multiple group comparisons

Lets say I have the following data frame ...
120 views

### What would be a good paper to cite to explain simulation methods in proc mixed in SAS?

In order to analyze data obtained from a repeated measure study, I used the following SAS code: ...
410 views

### Does using bootstrapping change how you deal with problems of Type I errors when testing multiple correlations?

I have been calculating p-values for correlations using bootstrapping. I've just been using the same p-value cutoff regardless of the number of pair-wise calculations. Intuitively, it seems to me that ...
292 views

### What software implements the multiple comparison sign test?

What software implements the multiple comparison sign test as described in Steel (1959)? References Steel, R.G.D. (1959). A multiple comparison sign test: treatments versus control. Journal of the ...
542 views

### Practical significance, especially with percents: “standard” measure and threshold

I, like many people, dislike statistical significance testing. I would much rather measure "practical significance" / effect size. The problem is that I do not know of a "standard" way of doing so. ...
318 views

### Is it acceptable to run two linear models on the same data set?

For a linear regression with multiple groups (natural groups defined a priori) is it acceptable to run two different models on the same data set to answer the following two questions? Does each ...
110 views

### Accurate comparison over time period

I have 52 samples daily, weekly and biweekly data for 3 subjects over the course of a 10 month period. At two points during the 10 month period the resolution increases to daily samples as the people ...
339 views

### Can I compare ordinal rankings (and if so, how)?

I have 15 supervisors who have ranked 21 employees from 1 (best) to 21 (worst), but the rankings are strictly ordinal-there is no interval data available. Can I compare any two (or more) supervisors' ...
422 views

### On the corrections for multiple comparisons

I've used the Mann-Whitney U-test in order to check for possible differences between two relatively small groups (12 subject in each) on 15 different items. I didn't have any pre-defined assumption ...
257 views

### Why use Bonferroni approximation for experiment-wise alpha?

It seems the Bonferroni method (dividing experimentwise alpha by # of comparisons) for choosing the p level to fix the experimentwise alpha (when doing many pairwise comparisons) is more conservative ...
3k views

### Similarity measures between curves?

I would like to compute the measure of similarity between two ordered sets of points---the ones under User compared with the ones under Teacher: The points are curves in 3D space, but I was ...
718 views

### Permutation tests in R for correlations

What's the easiest way to run permutation tests for a bivariate Pearson correlation matrix in R? I have tried running MPT.Corr, Multivariate Permutation Test for Correlations by Urbano Blackford et ...
2k views

### Bonferroni and Greenhouse-Geisser corrections with repeated-measures ANOVA

I guess my question comes in two parts: Following a 1x7 repeated-measures ANOVA, I carried out 3 pre-planned tests. Two were t-tests between 2 of the conditions. The third is a 1x6 ANOVA (to test ...
122 views

### How to interpret the T-depiction of multiple comparisons used in R's multcompView package?

R's multcompView package offers a seemingly useful set of tools to visualize the results of multiple comparisons, e.g. Tukey HSD analysis, including boxplots and 'letters' to identify ...
396 views

### Multiple comparisons process using Kruskal-Wallis, G-Test and corrections

I want to know if changing a button color has a positive effect on different measures (see the 4 measures below), using the following process: There's a default button, which is the fixed control ...
412 views

### Power of FDR vs. FWER approaches in multiple comparisons

Regarding multiple comparisons, could someone please explain me why the power of false discovery rate (FDR) is greater than the power of family-wise error rate (FWER)?
178 views

### Multiple FDR corrected experiments using the same data

I am testing many (500,000) genetic variants, and the tests are FDR corrected and give me a q-value. Normally I would just call everything with q < .05 significant. But in this case I am testing ...
214 views

### Overarching multiple comparison test for correlations?

I need to test for correlation of 4 sets of weather parameters between 2 sites. I am not interested in interactions between the parameters. Because no weather parameter is independent from other ...
808 views

### Experimentwise alpha for 2-way ANOVA

A two-way ANOVA yields 3 F ratios. Each F ratio has alpha of .05. Does that mean the experimentwise alpha level of the ANOVA is not .05 but rather is 1 - .95^3? If so, is there a recommended way to ...
605 views

### Reporting significance in Kruskal Wallis test

I performed the Kruskal-Wallis one-way ANOVA by ranks on five groups of data using Dataplot software. The software corrects for ties and uses Conover's procedure to do pairwise multiple comparisons. ...
11k views

### Significance of coefficients in linear regression: significant t-test vs non-significant F-statistic [duplicate]

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. R gives me the following p-values from the ...
206 views

### Sequential experiment: controlled design and metrics

This is a generic question. Let me put forth an example scenario. Say, I had 2 techniques for allocating my daily budget within 4 stocks. Upon allocation, I get the data about the stock's performance ...
150 views

### To what degree is FWE correction required for multiple permutation based t-tests

I am performing multiple t-tests to compare conditions of data. I have had to resort to permutation based t-tests (based on difference of means) due to high variance between samples. My question is ...
191 views

### Confidence interval for the biggest difference between means

I have a number of samples $X_1,\ldots,X_n$ from Gaussian distributions centered at $\mu_1,\ldots,\mu_n$ respectively. I need to construct a confidence interval for the biggest difference between ...
990 views

### Pairwise comparison of vectors with unequal sizes and unequal variances

I have 3 vectors which are unequal in size (lengths: 21, 33 and 7). Each vector contains ratio of bug-to-duplicate bugs in 3 different time periods. I have to find whether there is a difference ...
380 views

### Interaction term as a dependant variable in LMM with R

In a longitudinal study, two groups of subjects have been measured over a period of two years at 6 months intervals. During these measurements subjects have been assessed with a series of $k$ measures ...
181 views

### Multiple testing correction

I use glm.nb from MASS package over many observations. I then extract the p-values for the interaction term. Due to multiple ...
85 views

### Multiple measurements within groups test

I am measuring the X, Y, Z variables of i subjects in k states, and for each subject and state i am taking j measurements. What is the proper way to test for differences between the k states? It ...
278 views

### Determining the right method to compare observed versus measured data

I'm trying to solve this problem out loud. I'm not great at statistics but I try sometimes to understand where things are and where they're going. I'm trying to figure out the best method to compare ...
89 views

### Reducing the number of groups of data by joining them

I have one variable that attains a certain value (between 0 and 1), about 1000 observations and each belongs to one (and only one) of fifteen groups. Now I would like to reduce the number of groups to ...
2k views

### Why does one get a 0.000 Mann-Whitney U value (not p-value) etc.?

1) Is it okay to run a Mann-Whitney U test on unequal sample sizes i.e. two groups (N=11, N=12)or is there an alternative test? 2) What does it mean when one obtains a zero value for a Mann-Whitney U ...
1k views

### Cohen's d and multiple comparisons for 2/3-way ANOVA

I am conducting three-way ANOVA (A*B*C) with 2 levels each. 1) I found A*B interaction. 2) I moved to 2-way ANOVA (A*B) and found interaction again. I reported Eta-squared and equivalent Cohen's d. 3) ...
894 views

### Can one obtain a p-value for all levels of factor when doing a GLM?

Please help! I have recently been criticized for using pairwise comparisons to explain all three levels of a factor within a negative binomial GLM rather than all levels at once. I was told that it is ...
184 views

### How do I account for both intra- and inter-individual variance when comparing the control with a conditioned test stimulus?

I have data that consists of 10 repetitions of a measurement per person, for all repetitions there is one control value and one test value (after conditioning). This data will be reproduced in ...
1k views

### Is Benjamini-Hochberg correction more conservative as the number of comparisons increases?

How conservative is Benjamini-Hochberg multiple testing correction relative to the total number of comparisons? For example, if I have a list of 18,000 features for two groups and I perform a Wilcoxon ...
305 views

### Using the BH method for FDR control

I have preformed multiple tests and now have a list of all the calculated P-Values (~30K^2 values). I have a reason to believe my tests are somehow dependent. I would like to analyze the results and ...
148 views

### Alpha correction in a family of tests

Suppose that one runs 8 correlation analyses but in two separate batches: Would the alpha-correction apply to all 8 tests as a family or each of the 4 tests as a family? For example, if there are two ...
230 views

### Power analysis for matched poisson variables

I have two matched groups consisting of 22 observations each for a Poisson distributed variable. I am interested in evaluating the power to detect a 20% decrease in the outcome for the second group. I ...
2k views

### Alpha adjustment for multiple testing

I understand the logic of alpha adjustment for multiple testing. However, I am confused as to whether this correction should be applied to all tests on a dataset or only the pairwise comparison in ...