# Tagged Questions

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

222 views

### Permutation test for multiple correlation test statistics

I read some great discussions about using permutation tests on correlation matrices to deal with Type I errors that arise from the multiple comparisons; however I have a question about the correlation ...
867 views

### Comparing levels of factors after a GLM in R

Here is a little background about my situation: my data refer to the number of prey successfully eaten by a predator. As the number of prey is limited (25 available) in each trial, I had a column ...
113 views

### Purpose of Scheffé's confidence intervals

Scheffé's method, when first written about by Henry Scheffé, was described as serving the purpose of identifying statistically significant contrasts in multiple-comparison's ANOVA problems. But here ...
95 views

### What do “single-step” and “multi-step” mean in post-hoc testing of ANOVAs?

From Wikipedia Post-hoc testing of ANOVAs Multiple comparison procedures are commonly used in an analysis of variance after obtaining a significant omnibus test result, like the ANOVA ...
37 views

### Is the likelihood ratio test for the global null considered to be an omnibus test

Continue from my last question about an omnibus test. When the (global) null is the intersection of several individual nulls, I was wondering if the likelihood ratio test for the global null is ...
28 views

### Questions about large-scale multiple testing

From Wikipedia Traditional methods for multiple comparisons adjustments focus on correcting for modest numbers of comparisons, often in an analysis of variance. A different set of techniques ...
218 views

### Relation between omnibus test and multiple comparison?

Wikipedia says Methods which rely on an omnibus test before proceeding to multiple comparisons. Typically these methods require a significant ANOVA/Tukey's range test before proceeding to ...
56 views

### The definitions of Simultaneous inference and selective inference?

What are the definitions of Simultaneous inference and selective inference? I think they are two different ways of doing multiple comparison. I encountered them from Wikipedia, which I think only ...
111 views

### multiple regression and multiple comparisons

Say I fit a multiple regression of p explanatory variables. The t-test will allow me to check if any single one of those is significant ($H_0: \beta_i = 0$). I can do a partial F-test to check if some ...
121 views

### Statistically comparing results from two methods

I am not exactly sure if CrossValidated is the place to ask this question, but decided to give it a shot. This may be thought as an abstract question. I am trying to compare the feasibility of two ...
74 views

### Does false discovery rate estimate some population quantity?

False discovery rate (FDR) is defined as FDR = FP / (TP + FP). Does it estimate some population quantity, independent of sample, when the sample satisfies some condition? To make it clearer, ...
117 views

### When do Post-hoc tests require an adjustment for multiple comparisons

What happens to the Type 1 error rate, the Type 2 error rate, and Power if you do not adjust for multiple comparisons when you should? What happens to these as a result of using something like ...
55 views

### Using MatchIt to match groups in a retrospective analysis

I am interested in using the R package MatchIt to preprocess my data as to obtain matched groups based on a predefined treatment variable. However I am facing a few issues. The first issue is that ...
56 views

### How to choose q threshold for pFDR?

We are doing an activation map study and we want to control for false positives in multiple comparisons. Our map is around 5000 pixels. There's a lot of noise and variance so we see results that seem ...
31 views

### Gene expression statistical analysis

I need to confirm statistical difference for gene expression data, for two groups of data, standardized with different reference genes: ...
87 views

### Effect of corrections for multiple testing on sensitivity and specificity

Let's imagine we have one variable (factor.to.explain) that we want to explain by 10 other variables using 10 linear model (no interactions computed). We should correct for multiple testing. It is ...
48 views

### Is there a way to correct standard errors and/or prediction intervals for multiple comparison after doing backwards selection?

It is well known that most model selection algorithms can easily fall into a multiple comparison trap. To quote Friedman: Consider developing a regression model in a context where substantive ...
189 views

### Statistical test for multiple comparison with dependent observations (MATLAB)

I want to use matlab to perform a statistical test to find if there's a significantly higher mean return in any of n different sample populations. However, I can only find post hoc (although they can ...
148 views

### Do Bayes factors require multiple comparison correction?

As the title: Do Bayes factors require mutliple comparion correction? For more context, I am calculating very many likelihood ratio tests and I am thinking about how to handle multiple comparison ...
184 views

### Multiple hypothesis testing and F tests?

Does the multiple hypothesis testing problem apply to the calculation of an F statistic for joint significance? It seems to me that the more variables you are including in your test for joint ...
118 views

### How many chains should we choose in multi-treatment meta-analysis using MCMC and WinBugs

I am learning to undertake multiple treatment comparison meta-analysis (aka network meta-analysis or MTM) using WinBugs. One of the specifications are choosing the number of chains to run in the MCMC ...
21 views

### Correcting for multiple related correlations

We are using spearman correlations (non normally distributed data) to see if the amount of scarring found in an individuals heart (independent variable) is correlated with the amount of fractionation ...
26 views

### Multiple comparisons of multiple groups over multiple years

I have 14 years worth of data. For each year there are 5 possible payers for the trauma care delivered. I want to compare the significant changes of payers over time. Specifically that one of them ...
777 views

### Which test to use to compare proportions between 3 groups?

We are testing an e-mailing marketing campaign. On our initial test, we sent out two different e-mail types and had a third control group that did not receive an e-mail. Now we are getting back ...
119 views

### Analysis to use with categorical response and categorical predictor variables

I have data from a randomized survey experiment in which each respondent was assigned to one of 4 groups, one of which can be considered a "control" or "no treatment" group. The key question asked in ...
37 views

### Is it possible to predict/forecast results of pairwise comparison matrices based on one or two initial samples?

I'm curious if it's possible to predict the results of a pairwise comparison matrix that will be sent out to 100 people based on the results of one or two filled out surveys? The matrix has the ...
72 views

### Joint test of change in variance across matched groups

I am monitoring variability in sentencing outcomes (custodial sentence length) amongst matched groups. The aim is to determine whether sentence variability went up or down after a policy was ...
258 views

### Do we need a Bonferonni correction when running numerous univariate regression analyses?

For my data, I need to run numerous linear regression analyses, separately for each of my two groups of subjects as well as different conditions. Together, I would have to run approximately 40 ...
47 views

### Multiple t-tests

Suppose we have some baseline data $X_1$ with $n$ observations and some post-baseline data $X_2$ with $n$ observations. We want to compare whether the post-baseline data differs significantly in mean ...
83 views

### Comparing models of different types of ANOVA - multiple comparisons?

Statistical semi-novice question: Just following up this question, the fact that the three types of ANOVA exist with an unbalanced design, and that (presumably) you don't necessarily know a priori ...
108 views

### Multiple hypothesis tests, “comparing the expected values”

If I have two sets of random samples and I'd like to test whether there is some difference or not between them (or if one is better than the other) I'd test the null hypothesis $H_0: \mu_1 = \mu_2$ ...
41 views

### When to normalize learning?

I'm trying to determine the effect of three types of learning on a group of subjects. I have their pretest scores and posttest scores. The current goal is to determine which intervention reduce the ...
141 views

### Multiple comparison issue in multivariate logistic regression

I am assessing a bunch of risk factors related to HIV infection by logistic regression. First I will conduct a couple of chi square tests or t-tests to see if each factor is associated with HIV ...
27 views

### Mixfdr penalization parameter

I'm trying to use mixfdr in a simulation of p-values but sometimes the search for the penalization parameter spits out an error and shows some matrix entries to be infinite: ...
572 views

### Mean comparison for statistical significance

I was told to come here by a friend to ask a stats question. I took Social Stats and Methods of Social Research about 4 years ago and to be honest, I don't remember much about it. Now I'm writing a ...
31 views

### Detecting a trend in 4 samples

I have 4 levels for a single factor, which is called "Compliance". Compliance has an ordinal scale (0, 1, 2, 3) and it is computed as the number of rules a team complies with. My hypothesis is the ...
64 views

### Multiple Comparisons help

I have already asked a similar question but did not explain my problem in entirety. Any help would be greatly appreciated! I am analyzing some data to which I think I need to make some kind of ...
287 views

### How can I convert a q-value distribution to a p-value distribution?

Let's say I have a vector of q-values, which allow for handling multiple hypothesis testing by controlling the false discovery rate. Usually, these q-values will be generated from a distribution of ...
39 views

### Multiple comparisons in economic game

I am analysing some data to which I think I need to make some kind of multiple comparisons correction. The data comes from an economic game (Dictator game) and is the form of donations made by ...
928 views

I have two questions about Bonferroni adjustments: 1). Can one use the Bonferroni method to compare independent groups? The reason why I ask this is it seems that many examples I've encountered ...
210 views

### method of p value correction for a set of dependent p values

Here is my situation, I just got a list of 100 p-values, some of which are associated. For details, these 100 pvalues could be seperated into 7 groups, each group was calculated from a expression ...
72 views

### Sequential testing of multiple probabilities (binomial variables)

Task: I have e.g. 1 control group and 7 experimental groups which are treated differently. All groups have a dependent binomial variable: 1 = success, 0 = no success. I want to conduct sequential ...
147 views

### True meaning of $\pi_0 = 1$ in false discovery rate

I am not a statistician so apologies if my terminology is wrong I have a dataset of >14 million p values derived from Fisher's exact testing on genome scale sequencing data. Benjamini-Hochberg ...
54 views

### Comparing confidence limits

I have a data set that is broken into the four categories, each with four data points. I am asked to compare the confidence limits of the four categories. How do I go about doing this? I know to ...
239 views

### Comparing observed and predicted values across several measurements

As a neuropsychology graduate student with some experience in statistics (I'm usually the guy other psychologists come to with statistics problems after trying it themselves but before seeing a ...
81 views

### Test of differences (e.g. Hotelling's T) for aggregated data

I have 14 regression predictors and want to test statistically whether they differ from each other in their explained variance of the dependent variable. I decompose explained variance using the R ...
326 views

### Big Data vs multiple hypothesis testing?

Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics ...
859 views

### What is the correct way to choose tests for pairwise comparison after ANOVA?

I am measuring secretion of insulin upon glucose stimulation of pancreatic islets after various treatments. So I have two independent variables/factors: Treatment (three different compounds and a ...