Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
0answers
8 views

The definitions of Simultaneous inference and selective inference?

What are the definitions of Simultaneous inference and selective inference? I think they are two different ways of doing multiple comparison. I encountered them from Wikipedia, which I think only ...
1
vote
2answers
49 views

multiple regression and multiple comparisons

Say I fit a multiple regression of p explanatory variables. The t-test will allow me to check if any single one of those is significant ($H_0: \beta_i = 0$). I can do a partial F-test to check if some ...
0
votes
1answer
37 views

Statistically comparing results from two methods

I am not exactly sure if CrossValidated is the place to ask this question, but decided to give it a shot. This may be thought as an abstract question. I am trying to compare the feasibility of two ...
0
votes
0answers
12 views

lmer glht posthoc not giving pairwise comparisons

Trying to run posthoc analysis on lmer model. AOV shows significant difference between sex (three sexes) and position (four positions), both of which are factorial variables, and pl, which is ...
2
votes
1answer
65 views

Does false discovery rate estimate some population quantity?

False discovery rate (FDR) is defined as FDR = FP / (TP + FP). Does it estimate some population quantity, independent of sample, when the sample satisfies some condition? To make it clearer, ...
1
vote
3answers
48 views

When do Post-hoc tests require an adjustment for multiple comparisons

What happens to the Type 1 error rate, the Type 2 error rate, and Power if you do not adjust for multiple comparisons when you should? What happens to these as a result of using something like ...
1
vote
0answers
17 views

Using MatchIt to match groups in a retrospective analysis

I am interested in using the R package MatchIt to preprocess my data as to obtain matched groups based on a predefined treatment variable. However I am facing a few issues. The first issue is that ...
0
votes
1answer
16 views

How to choose q threshold for pFDR?

We are doing an activation map study and we want to control for false positives in multiple comparisons. Our map is around 5000 pixels. There's a lot of noise and variance so we see results that seem ...
0
votes
0answers
29 views

Gene expression statistical analysis

I need to confirm statistical difference for gene expression data, for two groups of data, standardized with different reference genes: ...
2
votes
1answer
44 views

Effect of corrections for multiple testing on sensitivity and specificity

Let's imagine we have one variable (factor.to.explain) that we want to explain by 10 other variables using 10 linear model (no interactions computed). We should correct for multiple testing. It is ...
1
vote
0answers
26 views

Is there a way to correct standard errors and/or prediction intervals for multiple comparison after doing backwards selection?

It is well known that most model selection algorithms can easily fall into a multiple comparison trap. To quote Friedman: Consider developing a regression model in a context where substantive ...
0
votes
0answers
52 views

Statistical test for multiple comparison with dependent observations (MATLAB)

I want to use matlab to perform a statistical test to find if there's a significantly higher mean return in any of n different sample populations. However, I can only find post hoc (although they can ...
2
votes
2answers
73 views

Do Bayes factors require multiple comparison correction?

As the title: Do Bayes factors require mutliple comparion correction? For more context, I am calculating very many likelihood ratio tests and I am thinking about how to handle multiple comparison ...
1
vote
1answer
70 views

Multiple hypothesis testing and F tests?

Does the multiple hypothesis testing problem apply to the calculation of an F statistic for joint significance? It seems to me that the more variables you are including in your test for joint ...
0
votes
0answers
38 views

How many chains should we choose in multi-treatment meta-analysis using MCMC and WinBugs

I am learning to undertake multiple treatment comparison meta-analysis (aka network meta-analysis or MTM) using WinBugs. One of the specifications are choosing the number of chains to run in the MCMC ...
0
votes
0answers
11 views

Correcting for multiple related correlations

We are using spearman correlations (non normally distributed data) to see if the amount of scarring found in an individuals heart (independent variable) is correlated with the amount of fractionation ...
0
votes
0answers
21 views

Multiple comparisons of multiple groups over multiple years

I have 14 years worth of data. For each year there are 5 possible payers for the trauma care delivered. I want to compare the significant changes of payers over time. Specifically that one of them ...
2
votes
2answers
143 views

Which test to use to compare proportions between 3 groups?

We are testing an e-mailing marketing campaign. On our initial test, we sent out two different e-mail types and had a third control group that did not receive an e-mail. Now we are getting back ...
0
votes
0answers
68 views

Analysis to use with categorical response and categorical predictor variables

I have data from a randomized survey experiment in which each respondent was assigned to one of 4 groups, one of which can be considered a "control" or "no treatment" group. The key question asked in ...
0
votes
1answer
32 views

Is it possible to predict/forecast results of pairwise comparison matrices based on one or two initial samples?

I'm curious if it's possible to predict the results of a pairwise comparison matrix that will be sent out to 100 people based on the results of one or two filled out surveys? The matrix has the ...
2
votes
1answer
44 views

Joint test of change in variance across matched groups

I am monitoring variability in sentencing outcomes (custodial sentence length) amongst matched groups. The aim is to determine whether sentence variability went up or down after a policy was ...
1
vote
2answers
114 views

Do we need a Bonferonni correction when running numerous univariate regression analyses?

For my data, I need to run numerous linear regression analyses, separately for each of my two groups of subjects as well as different conditions. Together, I would have to run approximately 40 ...
0
votes
1answer
42 views

Multiple t-tests

Suppose we have some baseline data $X_1$ with $n$ observations and some post-baseline data $X_2$ with $n$ observations. We want to compare whether the post-baseline data differs significantly in mean ...
4
votes
0answers
63 views

Comparing models of different types of ANOVA - multiple comparisons?

Statistical semi-novice question: Just following up this question, the fact that the three types of ANOVA exist with an unbalanced design, and that (presumably) you don't necessarily know a priori ...
0
votes
1answer
48 views

Multiple hypothesis tests, “comparing the expected values”

If I have two sets of random samples and I'd like to test whether there is some difference or not between them (or if one is better than the other) I'd test the null hypothesis $H_0: \mu_1 = \mu_2$ ...
2
votes
1answer
27 views

When to normalize learning?

I'm trying to determine the effect of three types of learning on a group of subjects. I have their pretest scores and posttest scores. The current goal is to determine which intervention reduce the ...
1
vote
1answer
62 views

Multiple comparison issue in multivariate logistic regression

I am assessing a bunch of risk factors related to HIV infection by logistic regression. First I will conduct a couple of chi square tests or t-tests to see if each factor is associated with HIV ...
0
votes
0answers
12 views

Mixfdr penalization parameter

I'm trying to use mixfdr in a simulation of p-values but sometimes the search for the penalization parameter spits out an error and shows some matrix entries to be infinite: ...
0
votes
2answers
173 views

Mean comparison for statistical significance

I was told to come here by a friend to ask a stats question. I took Social Stats and Methods of Social Research about 4 years ago and to be honest, I don't remember much about it. Now I'm writing a ...
1
vote
1answer
31 views

Detecting a trend in 4 samples

I have 4 levels for a single factor, which is called "Compliance". Compliance has an ordinal scale (0, 1, 2, 3) and it is computed as the number of rules a team complies with. My hypothesis is the ...
0
votes
1answer
52 views

Multiple Comparisons help

I have already asked a similar question but did not explain my problem in entirety. Any help would be greatly appreciated! I am analyzing some data to which I think I need to make some kind of ...
1
vote
1answer
102 views

How can I convert a q-value distribution to a p-value distribution?

Let's say I have a vector of q-values, which allow for handling multiple hypothesis testing by controlling the false discovery rate. Usually, these q-values will be generated from a distribution of ...
1
vote
0answers
37 views

Multiple comparisons in economic game

I am analysing some data to which I think I need to make some kind of multiple comparisons correction. The data comes from an economic game (Dictator game) and is the form of donations made by ...
1
vote
1answer
314 views

Two questions about Bonferroni adjustment

I have two questions about Bonferroni adjustments: 1). Can one use the Bonferroni method to compare independent groups? The reason why I ask this is it seems that many examples I've encountered ...
3
votes
1answer
125 views

method of p value correction for a set of dependent p values

Here is my situation, I just got a list of 100 p-values, some of which are associated. For details, these 100 pvalues could be seperated into 7 groups, each group was calculated from a expression ...
1
vote
0answers
45 views

Sequential testing of multiple probabilities (binomial variables)

Task: I have e.g. 1 control group and 7 experimental groups which are treated differently. All groups have a dependent binomial variable: 1 = success, 0 = no success. I want to conduct sequential ...
1
vote
1answer
115 views

True meaning of $\pi_0 = 1$ in false discovery rate

I am not a statistician so apologies if my terminology is wrong I have a dataset of >14 million p values derived from Fisher's exact testing on genome scale sequencing data. Benjamini-Hochberg ...
1
vote
0answers
46 views

Comparing confidence limits

I have a data set that is broken into the four categories, each with four data points. I am asked to compare the confidence limits of the four categories. How do I go about doing this? I know to ...
1
vote
0answers
145 views

Comparing observed and predicted values across several measurements

As a neuropsychology graduate student with some experience in statistics (I'm usually the guy other psychologists come to with statistics problems after trying it themselves but before seeing a ...
0
votes
0answers
51 views

Test of differences (e.g. Hotelling's T) for aggregated data

I have 14 regression predictors and want to test statistically whether they differ from each other in their explained variance of the dependent variable. I decompose explained variance using the R ...
4
votes
3answers
215 views

Big Data vs multiple hypothesis testing?

Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics ...
4
votes
2answers
220 views

What is the correct way to choose tests for pairwise comparison after ANOVA?

I am measuring secretion of insulin upon glucose stimulation of pancreatic islets after various treatments. So I have two independent variables/factors: Treatment (three different compounds and a ...
0
votes
1answer
39 views

Group sequential Design

Can anyone recommend a paper which provides a nice introduction or summary of group sequential design.
0
votes
1answer
135 views

Questions about the glht function for planned comparison

I have a nested model with the following effects fixed: treatments random: experiment_date I used lme() to model the data ...
2
votes
2answers
150 views

How to analyze categorical variables with multiple levels

I have 14 different habitat classes and two activity states (so two variables – habitat and activity). For activity state A, I have a data count of over 300, but with B, I only have around 30 (so ...
4
votes
1answer
192 views

Fisher overall p-value vs. pairwise comparisons

I am doing comparisons on response rates across three sites. Following are the cell counts ...
1
vote
1answer
102 views

Hypothesis testing on data with overlapping groups

I frequently run into the following example: Lot's of genes (order of hundreds of thousands), a few hundred samples stratified into multiple groups. And those groups overlap (i.e. healthy/sick + ...
1
vote
1answer
61 views

How to compare different heuristics over a large number of test instances

I have 3 different heuristics (H1, H2, H3) that are variation of genetic algorithms combined with some other techniques. The problem that I try to solve is The problem to minimize energy costs for ...
1
vote
1answer
133 views

Different types of error rates when there are multiple comparisons

I have seen 5 different type of error rates. They include: Componentwise Error Rate Experimentwise Error Rate False Discovery Rate Strong Familywise Error Rate Simultaneous Confidence Intervals ...
2
votes
0answers
32 views

Is this a valid method to control the FWER?

I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...

1 2 3 4 5