Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.
0
votes
0answers
8 views
The definitions of Simultaneous inference and selective inference?
What are the definitions of Simultaneous inference and selective inference? I think they are two different ways of doing multiple comparison. I encountered them from Wikipedia, which I think only ...
1
vote
2answers
49 views
multiple regression and multiple comparisons
Say I fit a multiple regression of p explanatory variables. The t-test will allow me to check if any single one of those is significant ($H_0: \beta_i = 0$). I can do a partial F-test to check if some ...
0
votes
1answer
37 views
Statistically comparing results from two methods
I am not exactly sure if CrossValidated is the place to ask this question, but decided to give it a shot. This may be thought as an abstract question. I am trying to compare the feasibility of two ...
0
votes
0answers
12 views
lmer glht posthoc not giving pairwise comparisons
Trying to run posthoc analysis on lmer model. AOV shows significant difference between sex (three sexes) and position (four positions), both of which are factorial variables, and pl, which is ...
2
votes
1answer
65 views
Does false discovery rate estimate some population quantity?
False discovery rate (FDR) is defined as
FDR = FP / (TP + FP).
Does it estimate some population quantity, independent of sample, when the sample satisfies some condition?
To make it clearer, ...
1
vote
3answers
48 views
When do Post-hoc tests require an adjustment for multiple comparisons
What happens to the Type 1 error rate, the Type 2 error rate, and Power if you do
not adjust for multiple comparisons when you should? What happens to these as a result
of using something like ...
1
vote
0answers
17 views
Using MatchIt to match groups in a retrospective analysis
I am interested in using the R package MatchIt to preprocess my data as to obtain matched groups based on a predefined treatment variable. However I am facing a few issues.
The first issue is that ...
0
votes
1answer
16 views
How to choose q threshold for pFDR?
We are doing an activation map study and we want to control for false positives in multiple comparisons. Our map is around 5000 pixels. There's a lot of noise and variance so we see results that seem ...
0
votes
0answers
29 views
Gene expression statistical analysis
I need to confirm statistical difference for gene expression data, for two groups of data, standardized with different reference genes:
...
2
votes
1answer
44 views
Effect of corrections for multiple testing on sensitivity and specificity
Let's imagine we have one variable (factor.to.explain) that we want to explain by 10 other variables using 10 linear model (no interactions computed). We should correct for multiple testing. It is ...
1
vote
0answers
26 views
Is there a way to correct standard errors and/or prediction intervals for multiple comparison after doing backwards selection?
It is well known that most model selection algorithms can easily fall into a multiple comparison trap. To quote Friedman:
Consider developing a regression model in a context where substantive
...
0
votes
0answers
52 views
Statistical test for multiple comparison with dependent observations (MATLAB)
I want to use matlab to perform a statistical test to find if there's a significantly higher mean return in any of n different sample populations. However, I can only find post hoc (although they can ...
2
votes
2answers
73 views
Do Bayes factors require multiple comparison correction?
As the title: Do Bayes factors require mutliple comparion correction?
For more context, I am calculating very many likelihood ratio tests and I am thinking about how to handle multiple comparison ...
1
vote
1answer
70 views
Multiple hypothesis testing and F tests?
Does the multiple hypothesis testing problem apply to the calculation of an F statistic for joint significance? It seems to me that the more variables you are including in your test for joint ...
0
votes
0answers
38 views
How many chains should we choose in multi-treatment meta-analysis using MCMC and WinBugs
I am learning to undertake multiple treatment comparison meta-analysis (aka network meta-analysis or MTM) using WinBugs. One of the specifications are choosing the number of chains to run in the MCMC ...
0
votes
0answers
11 views
Correcting for multiple related correlations
We are using spearman correlations (non normally distributed data) to see if the amount of scarring found in an individuals heart (independent variable) is correlated with the amount of fractionation ...
0
votes
0answers
21 views
Multiple comparisons of multiple groups over multiple years
I have 14 years worth of data. For each year there are 5 possible payers for the trauma care delivered. I want to compare the significant changes of payers over time. Specifically that one of them ...
2
votes
2answers
143 views
Which test to use to compare proportions between 3 groups?
We are testing an e-mailing marketing campaign. On our initial test, we sent out two different e-mail types and had a third control group that did not receive an e-mail. Now we are getting back ...
0
votes
0answers
68 views
Analysis to use with categorical response and categorical predictor variables
I have data from a randomized survey experiment in which each respondent was assigned to one of 4 groups, one of which can be considered a "control" or "no treatment" group. The key question asked in ...
0
votes
1answer
32 views
Is it possible to predict/forecast results of pairwise comparison matrices based on one or two initial samples?
I'm curious if it's possible to predict the results of a pairwise comparison matrix that will be sent out to 100 people based on the results of one or two filled out surveys?
The matrix has the ...
2
votes
1answer
44 views
Joint test of change in variance across matched groups
I am monitoring variability in sentencing outcomes (custodial sentence length) amongst matched groups. The aim is to determine whether sentence variability went up or down after a policy was ...
1
vote
2answers
114 views
Do we need a Bonferonni correction when running numerous univariate regression analyses?
For my data, I need to run numerous linear regression analyses, separately for each of my two groups of subjects as well as different conditions. Together, I would have to run approximately 40 ...
0
votes
1answer
42 views
Multiple t-tests
Suppose we have some baseline data $X_1$ with $n$ observations and some post-baseline data $X_2$ with $n$ observations. We want to compare whether the post-baseline data differs significantly in mean ...
4
votes
0answers
63 views
Comparing models of different types of ANOVA - multiple comparisons?
Statistical semi-novice question:
Just following up this question, the fact that the three types of ANOVA exist with an unbalanced design, and that (presumably) you don't necessarily know a priori ...
0
votes
1answer
48 views
Multiple hypothesis tests, “comparing the expected values”
If I have two sets of random samples and I'd like to test whether there is some difference or not between them (or if one is better than the other) I'd test the null hypothesis
$H_0: \mu_1 = \mu_2$ ...
2
votes
1answer
27 views
When to normalize learning?
I'm trying to determine the effect of three types of learning on a group of subjects.
I have their pretest scores and posttest scores.
The current goal is to determine which intervention reduce the ...
1
vote
1answer
62 views
Multiple comparison issue in multivariate logistic regression
I am assessing a bunch of risk factors related to HIV infection by logistic regression.
First I will conduct a couple of chi square tests or t-tests to see if each factor is associated with HIV ...
0
votes
0answers
12 views
Mixfdr penalization parameter
I'm trying to use mixfdr in a simulation of p-values but sometimes the search for the penalization parameter spits out an error and shows some matrix entries to be infinite:
...
0
votes
2answers
173 views
Mean comparison for statistical significance
I was told to come here by a friend to ask a stats question.
I took Social Stats and Methods of Social Research about 4 years ago and to be honest, I don't remember much about it. Now I'm writing a ...
1
vote
1answer
31 views
Detecting a trend in 4 samples
I have 4 levels for a single factor, which is called "Compliance".
Compliance has an ordinal scale (0, 1, 2, 3) and it is computed as the number of rules a team complies with.
My hypothesis is the ...
0
votes
1answer
52 views
Multiple Comparisons help
I have already asked a similar question but did not explain my problem in entirety. Any help would be greatly appreciated!
I am analyzing some data to which I think I need to make some kind of ...
1
vote
1answer
102 views
How can I convert a q-value distribution to a p-value distribution?
Let's say I have a vector of q-values, which allow for handling multiple hypothesis testing by controlling the false discovery rate. Usually, these q-values will be generated from a distribution of ...
1
vote
0answers
37 views
Multiple comparisons in economic game
I am analysing some data to which I think I need to make some kind of multiple comparisons correction.
The data comes from an economic game (Dictator game) and is the form of donations made by ...
1
vote
1answer
314 views
Two questions about Bonferroni adjustment
I have two questions about Bonferroni adjustments:
1). Can one use the Bonferroni method to compare independent groups? The reason why I ask this is it seems that many examples I've encountered ...
3
votes
1answer
125 views
method of p value correction for a set of dependent p values
Here is my situation, I just got a list of 100 p-values, some of which are associated.
For details, these 100 pvalues could be seperated into 7 groups, each group was calculated from a expression ...
1
vote
0answers
45 views
Sequential testing of multiple probabilities (binomial variables)
Task:
I have e.g. 1 control group and 7 experimental groups which are treated differently. All groups have a dependent binomial variable: 1 = success, 0 = no success.
I want to conduct sequential ...
1
vote
1answer
115 views
True meaning of $\pi_0 = 1$ in false discovery rate
I am not a statistician so apologies if my terminology is wrong
I have a dataset of >14 million p values derived from Fisher's exact testing on genome scale sequencing data.
Benjamini-Hochberg ...
1
vote
0answers
46 views
Comparing confidence limits
I have a data set that is broken into the four categories, each with four data points. I am asked to compare the confidence limits of the four categories. How do I go about doing this?
I know to ...
1
vote
0answers
145 views
Comparing observed and predicted values across several measurements
As a neuropsychology graduate student with some experience in statistics (I'm usually the guy other psychologists come to with statistics problems after trying it themselves but before seeing a ...
0
votes
0answers
51 views
Test of differences (e.g. Hotelling's T) for aggregated data
I have 14 regression predictors and want to test statistically whether they differ from each other in their explained variance of the dependent variable. I decompose explained variance using the R ...
4
votes
3answers
215 views
Big Data vs multiple hypothesis testing?
Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics ...
4
votes
2answers
220 views
What is the correct way to choose tests for pairwise comparison after ANOVA?
I am measuring secretion of insulin upon glucose stimulation of pancreatic islets after various treatments.
So I have two independent variables/factors:
Treatment (three different compounds and a ...
0
votes
1answer
39 views
Group sequential Design
Can anyone recommend a paper which provides a nice introduction or summary of group sequential design.
0
votes
1answer
135 views
Questions about the glht function for planned comparison
I have a nested model with the following effects
fixed: treatments
random: experiment_date
I used lme() to model the data
...
2
votes
2answers
150 views
How to analyze categorical variables with multiple levels
I have 14 different habitat classes and two activity states (so two variables – habitat and activity). For activity state A, I have a data count of over 300, but with B, I only have around 30 (so ...
4
votes
1answer
192 views
Fisher overall p-value vs. pairwise comparisons
I am doing comparisons on response rates across three sites. Following are the cell counts
...
1
vote
1answer
102 views
Hypothesis testing on data with overlapping groups
I frequently run into the following example:
Lot's of genes (order of hundreds of thousands), a few hundred samples stratified into multiple groups. And those groups overlap (i.e. healthy/sick + ...
1
vote
1answer
61 views
How to compare different heuristics over a large number of test instances
I have 3 different heuristics (H1, H2, H3) that are variation of genetic algorithms combined with some other techniques.
The problem that I try to solve is The problem to minimize energy costs for ...
1
vote
1answer
133 views
Different types of error rates when there are multiple comparisons
I have seen 5 different type of error rates. They include:
Componentwise Error Rate
Experimentwise Error Rate
False Discovery Rate
Strong Familywise Error Rate
Simultaneous Confidence Intervals
...
2
votes
0answers
32 views
Is this a valid method to control the FWER?
I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
