Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.
3
votes
2answers
379 views
Following up a three-way interaction found with (mixed) model comparison
I hope this isn't a silly question, I'd like some advice on following up a threeway interaction in a mixed effects model. I've been building my models incrementally, like this:
...
2
votes
2answers
267 views
p-value adjustment via permutation for multiple correlations
I have the following problem: I have performed 3200 correlations between a certain measure of connectivity and 1 behavioural outcome in a sample of subjects. As the single correlations (represented by ...
3
votes
0answers
234 views
3
votes
1answer
104 views
What would be a good paper to cite to explain simulation methods in proc mixed in SAS?
In order to analyze data obtained from a repeated measure study, I used the following SAS code:
...
4
votes
1answer
293 views
Does using bootstrapping change how you deal with problems of Type I errors when testing multiple correlations?
I have been calculating p-values for correlations using bootstrapping. I've just been using the same p-value cutoff regardless of the number of pair-wise calculations. Intuitively, it seems to me that ...
2
votes
0answers
237 views
What software implements the multiple comparison sign test?
What software implements the multiple comparison sign test as described in Steel (1959)?
References
Steel, R.G.D. (1959). A multiple comparison sign test: treatments versus control. Journal of the ...
4
votes
4answers
327 views
Practical significance, especially with percents: “standard” measure and threshold
I, like many people, dislike statistical significance testing. I would much rather measure "practical significance" / effect size.
The problem is that I do not know of a "standard" way of doing so. ...
8
votes
1answer
272 views
Is it acceptable to run two linear models on the same data set?
For a linear regression with multiple groups (natural groups defined a priori) is it acceptable to run two different models on the same data set to answer the following two questions?
Does each ...
0
votes
0answers
90 views
Accurate comparison over time period
I have 52 samples daily, weekly and biweekly data for 3 subjects over the course of a 10 month period. At two points during the 10 month period the resolution increases to daily samples as the people ...
4
votes
2answers
192 views
Can I compare ordinal rankings (and if so, how)?
I have 15 supervisors who have ranked 21 employees from 1 (best) to 21 (worst), but the rankings are strictly ordinal-there is no interval data available. Can I compare any two (or more) supervisors' ...
3
votes
2answers
292 views
On the corrections for multiple comparisons
I've used the Mann-Whitney U-test in order to check for possible differences between two relatively small groups (12 subject in each) on 15 different items. I didn't have any pre-defined assumption ...
2
votes
2answers
224 views
Why use Bonferroni approximation for experiment-wise alpha?
It seems the Bonferroni method (dividing experimentwise alpha by # of comparisons) for choosing the p level to fix the experimentwise alpha (when doing many pairwise comparisons) is more conservative ...
7
votes
5answers
1k views
Similarity measures between curves?
I would like to compute the measure of similarity between two ordered sets of points---the ones under User compared with the ones under Teacher:
The points are curves in 3D space, but I was ...
1
vote
1answer
533 views
Permutation tests in R for correlations
What's the easiest way to run permutation tests for a bivariate Pearson correlation matrix in R?
I have tried running MPT.Corr, Multivariate Permutation Test
for Correlations by Urbano Blackford et ...
4
votes
2answers
1k views
Bonferroni and Greenhouse-Geisser corrections with repeated-measures ANOVA
I guess my question comes in two parts:
Following a 1x7 repeated-measures ANOVA, I carried out 3 pre-planned tests. Two were t-tests between 2 of the conditions. The third is a 1x6 ANOVA (to test ...
1
vote
1answer
87 views
How to interpret the T-depiction of multiple comparisons used in R's multcompView package?
R's multcompView package offers a seemingly useful set of tools to visualize the results of multiple comparisons, e.g. Tukey HSD analysis, including boxplots and 'letters' to identify ...
1
vote
0answers
292 views
Multiple comparisons process using Kruskal-Wallis, G-Test and corrections
I want to know if changing a button color has a positive effect on different measures (see the 4 measures below), using the following process:
There's a default button, which is the fixed control ...
5
votes
1answer
274 views
Power of FDR vs. FWER approaches in multiple comparisons
Regarding multiple comparisons, could someone please explain me why the power of false discovery rate (FDR) is greater than the power of family-wise error rate (FWER)?
6
votes
1answer
131 views
Multiple FDR corrected experiments using the same data
I am testing many (500,000) genetic variants, and the tests are FDR corrected and give me a q-value. Normally I would just call everything with q < .05 significant.
But in this case I am testing ...
1
vote
1answer
170 views
Overarching multiple comparison test for correlations?
I need to test for correlation of 4 sets of weather parameters between 2 sites. I am not interested in interactions between the parameters. Because no weather parameter is independent from other ...
3
votes
2answers
536 views
Experimentwise alpha for 2-way ANOVA
A two-way ANOVA yields 3 F ratios. Each F ratio has alpha of .05. Does that mean the experimentwise alpha level of the ANOVA is not .05 but rather is 1 - .95^3? If so, is there a recommended way to ...
2
votes
0answers
442 views
Reporting significance in Kruskal Wallis test
I performed the Kruskal-Wallis one-way ANOVA by ranks on five groups of data using Dataplot software. The software corrects for ties and uses Conover's procedure to do pairwise multiple comparisons.
...
0
votes
4answers
308 views
How to measure the degree of overlapping between two sets of distribution? [closed]
I have four sets of distribution $X_1$, $X_2$, $Y_1$ and $Y_2$.
I would like to compare which pair is more overlapping: $X_1$ vs. $Y_1$, or $X_2$ vs. $Y_2$.
Is there any test or quantifier for me to ...
8
votes
3answers
8k views
Significance of coefficients in linear regression: significant t-test vs non-significant F-statistic [duplicate]
I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations.
R gives me the following p-values from the ...
1
vote
1answer
177 views
Sequential experiment: controlled design and metrics
This is a generic question. Let me put forth an example scenario. Say, I had 2 techniques for allocating my daily budget within 4 stocks. Upon allocation, I get the data about the stock's performance ...
1
vote
2answers
102 views
To what degree is FWE correction required for multiple permutation based t-tests
I am performing multiple t-tests to compare conditions of data. I have had to resort to permutation based t-tests (based on difference of means) due to high variance between samples.
My question is ...
3
votes
0answers
177 views
Confidence interval for the biggest difference between means
I have a number of samples $X_1,\ldots,X_n $ from Gaussian distributions centered at $\mu_1,\ldots,\mu_n $ respectively.
I need to construct a confidence interval for the biggest difference between ...
1
vote
2answers
679 views
Pairwise comparison of vectors with unequal sizes and unequal variances
I have 3 vectors which are unequal in size (lengths: 21, 33 and 7). Each vector contains ratio of bug-to-duplicate bugs in 3 different time periods. I have to find whether there is a difference ...
1
vote
2answers
315 views
Interaction term as a dependant variable in LMM with R
In a longitudinal study, two groups of subjects have been measured over a period of two years at 6 months intervals. During these measurements subjects have been assessed with a series of $k$ measures ...
1
vote
1answer
162 views
Multiple testing correction
I use glm.nb from MASS package over many observations. I then extract the p-values for the interaction term. Due to multiple ...
0
votes
1answer
79 views
Multiple measurements within groups test
I am measuring the X, Y, Z variables of i subjects in k states, and for each subject and state i am taking j measurements. What is the proper way to test for differences between the k states?
It ...
4
votes
1answer
258 views
Determining the right method to compare observed versus measured data
I'm trying to solve this problem out loud. I'm not great at statistics but I try sometimes to understand where things are and where they're going.
I'm trying to figure out the best method to compare ...
0
votes
1answer
86 views
Reducing the number of groups of data by joining them
I have one variable that attains a certain value (between 0 and 1), about 1000 observations and each belongs to one (and only one) of fifteen groups. Now I would like to reduce the number of groups to ...
0
votes
0answers
1k views
Why does one get a 0.000 Mann-Whitney U value (not p-value) etc.?
1) Is it okay to run a Mann-Whitney U test on unequal sample sizes i.e. two groups (N=11, N=12)or is there an alternative test?
2) What does it mean when one obtains a zero value for a Mann-Whitney U ...
2
votes
1answer
971 views
Cohen's d and multiple comparisons for 2/3-way ANOVA
I am conducting three-way ANOVA (A*B*C) with 2 levels each.
1) I found A*B interaction.
2) I moved to 2-way ANOVA (A*B) and found interaction again. I reported Eta-squared and equivalent Cohen's d.
3) ...
0
votes
1answer
659 views
Can one obtain a p-value for all levels of factor when doing a GLM?
Please help!
I have recently been criticized for using pairwise comparisons to explain all three levels of a factor within a negative binomial GLM rather than all levels at once. I was told that it is ...
2
votes
1answer
141 views
How do I account for both intra- and inter-individual variance when comparing the control with a conditioned test stimulus?
I have data that consists of 10 repetitions of a measurement per person, for all repetitions there is one control value and one test value (after conditioning). This data will be reproduced in ...
3
votes
1answer
714 views
Is Benjamini-Hochberg correction more conservative as the number of comparisons increases?
How conservative is Benjamini-Hochberg multiple testing correction relative to the total number of comparisons? For example, if I have a list of 18,000 features for two groups and I perform a Wilcoxon ...
1
vote
1answer
249 views
Using the BH method for FDR control
I have preformed multiple tests and now have a list of all the calculated P-Values (~30K^2 values). I have a reason to believe my tests are somehow dependent.
I would like to analyze the results and ...
1
vote
1answer
142 views
Alpha correction in a family of tests
Suppose that one runs 8 correlation analyses but in two separate batches: Would the alpha-correction apply to all 8 tests as a family or each of the 4 tests as a family?
For example, if there are two ...
4
votes
1answer
188 views
Power analysis for matched poisson variables
I have two matched groups consisting of 22 observations each for a Poisson distributed variable. I am interested in evaluating the power to detect a 20% decrease in the outcome for the second group. I ...
5
votes
3answers
1k views
Alpha adjustment for multiple testing
I understand the logic of alpha adjustment for multiple testing. However, I am confused as to whether this correction should be applied to all tests on a dataset or only the pairwise comparison in ...
2
votes
1answer
126 views
What is the best single measure of brain networks based on functional MRI data?
I want to analyze functional MRI (fMRI) data as follows:
I am comparing the brain networks of two groups of subjects (patients and matched).
For each subject I have a correlation (Pearson) matrix ...
3
votes
0answers
162 views
How to determine multivariate confidence in laymans terms
I am trying to determine colors to best grab a web-users attention.
I have a simple system which randomly chooses one of 4 foreground colors, and one of 6 background colors.
I then save the stats in ...
1
vote
0answers
471 views
Fisher's Exact Test and large tables with pairwise
I found that it is possible to look at adjusted residuals and treat them as a z-score and then use a two-tailed test to find a p-value. The problem is how do you calculate adjusted residuals on what I ...
3
votes
2answers
235 views
What might be a clear, practical definition for a “family of hypotheses” (with respect to familywise error rate)?
When trying to evaluate what constitutes a family of hypotheses within an experiment/project/analysis, I've found "similar in purpose" and "similar in content" given as guidelines for delimiting ...
4
votes
2answers
794 views
Multiple comparisons with ANOVA including one between and one within-subject effect
I'm trying to look at multiple comparisons (across levels of my between subjects factor) for a model with one between-subject factor and one within-subject factor. I was trying to use TukeyHSD on an ...
6
votes
1answer
184 views
What kind of statistical method do I need for comparing element concentrations?
I have assayed the concentrations of various metal elements in
different anatomic regions of two strains of mice. Now, for each
element, in each region, I want to find whether there
is any difference ...
4
votes
1answer
97 views
Mean of one group vs. the rest of the sample
I have data from a survey with Likert-type questions (Rate how strongly you feel about this issue on a scale from 1 to 10).
We know the respondents home town (as well as some other information, such ...
5
votes
2answers
469 views
Understanding MANOVA in case of a single predictor
I'm trying to understand the statistical analysis I saw in a clinical study. They measured performance of 3 groups of subjects with a series of performance measures (A, B, C, ..., N). The objective ...
