Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.
1
vote
3answers
37 views
When do Post-hoc tests require an adjustment for multiple comparisons
What happens to the Type 1 error rate, the Type 2 error rate, and Power if you do
not adjust for multiple comparisons when you should? What happens to these as a result
of using something like ...
1
vote
0answers
16 views
Using MatchIt to match groups in a retrospective analysis
I am interested in using the R package MatchIt to preprocess my data as to obtain matched groups based on a predefined treatment variable. However I am facing a few issues.
The first issue is that ...
0
votes
1answer
14 views
How to choose q threshold for pFDR?
We are doing an activation map study and we want to control for false positives in multiple comparisons. Our map is around 5000 pixels. There's a lot of noise and variance so we see results that seem ...
0
votes
0answers
29 views
Gene expression statistical analysis
I need to confirm statistical difference for gene expression data, for two groups of data, standardized with different reference genes:
...
2
votes
1answer
44 views
Effect of corrections for multiple testing on sensitivity and specificity
Let's imagine we have one variable (factor.to.explain) that we want to explain by 10 other variables using 10 linear model (no interactions computed). We should correct for multiple testing. It is ...
1
vote
0answers
25 views
Is there a way to correct standard errors and/or prediction intervals for multiple comparison after doing backwards selection?
It is well known that most model selection algorithms can easily fall into a multiple comparison trap. To quote Friedman:
Consider developing a regression model in a context where substantive
...
0
votes
0answers
42 views
Statistical test for multiple comparison with dependent observations (MATLAB)
I want to use matlab to perform a statistical test to find if there's a significantly higher mean return in any of n different sample populations. However, I can only find post hoc (although they can ...
2
votes
2answers
71 views
Do Bayes factors require multiple comparison correction?
As the title: Do Bayes factors require mutliple comparion correction?
For more context, I am calculating very many likelihood ratio tests and I am thinking about how to handle multiple comparison ...
1
vote
1answer
67 views
Multiple hypothesis testing and F tests?
Does the multiple hypothesis testing problem apply to the calculation of an F statistic for joint significance? It seems to me that the more variables you are including in your test for joint ...
0
votes
0answers
33 views
How many chains should we choose in multi-treatment meta-analysis using MCMC and WinBugs
I am learning to undertake multiple treatment comparison meta-analysis (aka network meta-analysis or MTM) using WinBugs. One of the specifications are choosing the number of chains to run in the MCMC ...
0
votes
0answers
11 views
Correcting for multiple related correlations
We are using spearman correlations (non normally distributed data) to see if the amount of scarring found in an individuals heart (independent variable) is correlated with the amount of fractionation ...
0
votes
0answers
21 views
Multiple comparisons of multiple groups over multiple years
I have 14 years worth of data. For each year there are 5 possible payers for the trauma care delivered. I want to compare the significant changes of payers over time. Specifically that one of them ...
2
votes
2answers
126 views
Which test to use to compare proportions between 3 groups?
We are testing an e-mailing marketing campaign. On our initial test, we sent out two different e-mail types and had a third control group that did not receive an e-mail. Now we are getting back ...
0
votes
0answers
64 views
Analysis to use with categorical response and categorical predictor variables
I have data from a randomized survey experiment in which each respondent was assigned to one of 4 groups, one of which can be considered a "control" or "no treatment" group. The key question asked in ...
0
votes
1answer
32 views
Is it possible to predict/forecast results of pairwise comparison matrices based on one or two initial samples?
I'm curious if it's possible to predict the results of a pairwise comparison matrix that will be sent out to 100 people based on the results of one or two filled out surveys?
The matrix has the ...
2
votes
1answer
44 views
Joint test of change in variance across matched groups
I am monitoring variability in sentencing outcomes (custodial sentence length) amongst matched groups. The aim is to determine whether sentence variability went up or down after a policy was ...
1
vote
2answers
113 views
Do we need a Bonferonni correction when running numerous univariate regression analyses?
For my data, I need to run numerous linear regression analyses, separately for each of my two groups of subjects as well as different conditions. Together, I would have to run approximately 40 ...
0
votes
1answer
42 views
Multiple t-tests
Suppose we have some baseline data $X_1$ with $n$ observations and some post-baseline data $X_2$ with $n$ observations. We want to compare whether the post-baseline data differs significantly in mean ...
4
votes
0answers
61 views
Comparing models of different types of ANOVA - multiple comparisons?
Statistical semi-novice question:
Just following up this question, the fact that the three types of ANOVA exist with an unbalanced design, and that (presumably) you don't necessarily know a priori ...
0
votes
1answer
46 views
Multiple hypothesis tests, “comparing the expected values”
If I have two sets of random samples and I'd like to test whether there is some difference or not between them (or if one is better than the other) I'd test the null hypothesis
$H_0: \mu_1 = \mu_2$ ...
2
votes
1answer
27 views
When to normalize learning?
I'm trying to determine the effect of three types of learning on a group of subjects.
I have their pretest scores and posttest scores.
The current goal is to determine which intervention reduce the ...
1
vote
1answer
59 views
Multiple comparison issue in multivariate logistic regression
I am assessing a bunch of risk factors related to HIV infection by logistic regression.
First I will conduct a couple of chi square tests or t-tests to see if each factor is associated with HIV ...
0
votes
0answers
12 views
Mixfdr penalization parameter
I'm trying to use mixfdr in a simulation of p-values but sometimes the search for the penalization parameter spits out an error and shows some matrix entries to be infinite:
...
0
votes
2answers
163 views
Mean comparison for statistical significance
I was told to come here by a friend to ask a stats question.
I took Social Stats and Methods of Social Research about 4 years ago and to be honest, I don't remember much about it. Now I'm writing a ...
1
vote
1answer
30 views
Detecting a trend in 4 samples
I have 4 levels for a single factor, which is called "Compliance".
Compliance has an ordinal scale (0, 1, 2, 3) and it is computed as the number of rules a team complies with.
My hypothesis is the ...
0
votes
1answer
51 views
Multiple Comparisons help
I have already asked a similar question but did not explain my problem in entirety. Any help would be greatly appreciated!
I am analyzing some data to which I think I need to make some kind of ...
1
vote
1answer
100 views
How can I convert a q-value distribution to a p-value distribution?
Let's say I have a vector of q-values, which allow for handling multiple hypothesis testing by controlling the false discovery rate. Usually, these q-values will be generated from a distribution of ...
1
vote
0answers
36 views
Multiple comparisons in economic game
I am analysing some data to which I think I need to make some kind of multiple comparisons correction.
The data comes from an economic game (Dictator game) and is the form of donations made by ...
1
vote
1answer
288 views
Two questions about Bonferroni adjustment
I have two questions about Bonferroni adjustments:
1). Can one use the Bonferroni method to compare independent groups? The reason why I ask this is it seems that many examples I've encountered ...
3
votes
1answer
123 views
method of p value correction for a set of dependent p values
Here is my situation, I just got a list of 100 p-values, some of which are associated.
For details, these 100 pvalues could be seperated into 7 groups, each group was calculated from a expression ...
1
vote
0answers
44 views
Sequential testing of multiple probabilities (binomial variables)
Task:
I have e.g. 1 control group and 7 experimental groups which are treated differently. All groups have a dependent binomial variable: 1 = success, 0 = no success.
I want to conduct sequential ...
1
vote
1answer
113 views
True meaning of $\pi_0 = 1$ in false discovery rate
I am not a statistician so apologies if my terminology is wrong
I have a dataset of >14 million p values derived from Fisher's exact testing on genome scale sequencing data.
Benjamini-Hochberg ...
1
vote
0answers
46 views
Comparing confidence limits
I have a data set that is broken into the four categories, each with four data points. I am asked to compare the confidence limits of the four categories. How do I go about doing this?
I know to ...
1
vote
0answers
139 views
Comparing observed and predicted values across several measurements
As a neuropsychology graduate student with some experience in statistics (I'm usually the guy other psychologists come to with statistics problems after trying it themselves but before seeing a ...
0
votes
0answers
51 views
Test of differences (e.g. Hotelling's T) for aggregated data
I have 14 regression predictors and want to test statistically whether they differ from each other in their explained variance of the dependent variable. I decompose explained variance using the R ...
4
votes
3answers
209 views
Big Data vs multiple hypothesis testing?
Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics ...
4
votes
2answers
213 views
What is the correct way to choose tests for pairwise comparison after ANOVA?
I am measuring secretion of insulin upon glucose stimulation of pancreatic islets after various treatments.
So I have two independent variables/factors:
Treatment (three different compounds and a ...
0
votes
1answer
38 views
Group sequential Design
Can anyone recommend a paper which provides a nice introduction or summary of group sequential design.
0
votes
1answer
131 views
Questions about the glht function for planned comparison
I have a nested model with the following effects
fixed: treatments
random: experiment_date
I used lme() to model the data
...
2
votes
2answers
149 views
How to analyze categorical variables with multiple levels
I have 14 different habitat classes and two activity states (so two variables – habitat and activity). For activity state A, I have a data count of over 300, but with B, I only have around 30 (so ...
4
votes
1answer
186 views
Fisher overall p-value vs. pairwise comparisons
I am doing comparisons on response rates across three sites. Following are the cell counts
...
1
vote
1answer
102 views
Hypothesis testing on data with overlapping groups
I frequently run into the following example:
Lot's of genes (order of hundreds of thousands), a few hundred samples stratified into multiple groups. And those groups overlap (i.e. healthy/sick + ...
1
vote
1answer
60 views
How to compare different heuristics over a large number of test instances
I have 3 different heuristics (H1, H2, H3) that are variation of genetic algorithms combined with some other techniques.
The problem that I try to solve is The problem to minimize energy costs for ...
1
vote
1answer
132 views
Different types of error rates when there are multiple comparisons
I have seen 5 different type of error rates. They include:
Componentwise Error Rate
Experimentwise Error Rate
False Discovery Rate
Strong Familywise Error Rate
Simultaneous Confidence Intervals
...
2
votes
0answers
31 views
Is this a valid method to control the FWER?
I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
1
vote
0answers
71 views
Model comparison across multiple correlated responses
I have two multivariate linear regression models (multiple outcomes, i.e., the responses are a matrix), and I'm measuring their performance using $R^2$ in cross-validation, over these individual ...
1
vote
0answers
46 views
3 groups of 8 with a different treatment each, over 45 minutes - confused as to stats test needed
I have three experimental groups. Each will provide a 15 minute baseline before an addition of either drug A, drug B or a saline control. Each group has an N of 8 as maximum, but possibly as low as 6 ...
0
votes
0answers
27 views
What test to perform to obtain measure of multivariate stability
Oversimplified example for purpose of explanation of the problem:
Imagine you have an exam with 2 questions of different severity, each question gives 10 points - 20 points overall. You run the test ...
2
votes
1answer
506 views
Mixed model multiple comparisons for interaction between continuous and categorical predictor
I would like to use lme4 to fit a mixed effects regression and multcomp to compute the pairwise comparisons. I have a complex ...
4
votes
1answer
170 views
Is it necessary to correct alpha in repeated measures ANOVA before any post-hoc comparisons?
I have a cross-over design where a sample of subjects undergoes 7 different interventions, and within each intervention there are 3 time points (pre-intervention, post-intervention1, ...
