# Tagged Questions

27 views

### Hypothesis testing for more than two samples?

We do a monthly fundraising letter at my nonprofit. I know how to set up tests in R to compare whether the proportion of response and amount received between two mailings is significantly different, ...
30 views

### When designing experiments why is it necessary to have multiple groups run in parallel?

Let's say I wish to test the effectiveness of a new technique of teaching as opposed to an old one. Why is it recommended to have a control group running the experiment in parallel with the group ...
95 views

### Combining p-values from different statistical tests applied on the same data

Although the title of the question appears trivial, I would like to explain that it is not that trivial in the sense that it's different from the question of applying the same statistical test in ...
43 views

### Multiple hypothesis testing of same hypothesis

I read an interesting statement: running the same data through multiple tests no longer counts as multiple comparisons. I am fairly confident this is incorrect if you are testing different ...
132 views

### What is the best way to analyze data if your experimental design changes while running the experiment?

I imagine this is a somewhat common situation in practice. I am thinking mainly in terms of preclinical drug trials. 1) During the course of the study a new technique is learned or some time/money is ...
23 views

### “Repeated measures” when the measures are not independent?

I have a set of data consisting of measurements in ten subjects of the same physiological parameters. The measurement and processing of the data have been carried out at two sites, A and B. For each ...
21 views

### One experiment with many (dependent!) hypotheses for motif prediction in dna sequences

I am having the following problem. I have a group of words which are all permutations of each other. On a random dataset of DNA sequences I can calculate the distribution of the number of word ...
148 views

### What are Hommel Hochberg corrections?

I have recently been introduced to to Hommel Hochberg corrections. I am trying to find a simple explanation about what this actually is/does, but am having no luck. Can anyone please give a brief and ...
24 views

### How to calculate the degree of freedom for Williams' Test

Williams' test is a multiple comparison test. My question is, when compare the Williams' test statistic with the tabulated critical values, which degree of freedom should I look for. In the SAS ...
40 views

### Multiple comparisons?

I have four groups of entities; each one of the groups is subject to three treatments (two known ones and one experimental); results of each treatment are measured using two related measurement ...
58 views

### Comparing the mean of several exponentially distributed variables

(Possibly related to Is there a nonparametric equivalent of Tukey HSD?) Given a set of several exponentially distributed variables (representing data which is modeled this way), I'd like to check ...
85 views

### Is analysis of existing data always exploratory, or can it be used for hypothesis testing?

This is a question about the rhetoric of describing analysis done using a public data set or any other pre-existing data set. Here is the hypothetical situation: A researcher reports that they have a ...
48 views

### Normalization to control

Given two experimental runs, each with an experimental & a control group, I would like to test for a statistically significant difference between the first and second experimental group (using ...
55 views

### Why in this case do the Dunnett's p values are barely any higher than the ones of an unpaired t test?

I was looking at an example from Professor David Howell from the University of Vermont (who I think has recently retired) regarding Post Hoc and Planned Comparison tests. His example entails ...
72 views

### Multiple testing, FDR and power

Suppose that I've a data with a total of 30,000 observations. Each observation has a total of 6 values, 3 per condition. I'm interested in, let's say, testing if the means of these two conditions for ...
26 views

### What is the formula for calculating the Simple Contrasts test for a Within-Groups ANOVA?

I need to be able to calculate long hand some Planned Comparison test for a Within-Groups ANOVA framework. I understand the most common tests in such a situation are "Contrasts" test including: ...
52 views

### Hypothesis test for summary data involving percentages

500 patients had an operation to remove stomach cancer. There exists a clinical guideline that suggests that a minimum of ten lymph nodes should be removed during the operation. I am therefore ...
196 views

### Why use ANOVA at all instead of jumping straight into post-hoc or planned comparisons tests?

Looking at a between-groups ANOVA situation, what do you get by actually doing such an ANOVA test first, and second do post-hoc (Bonferroni, Šidák, etc.) or planned comparisons tests? Why not skip the ...
108 views

### Could I use Tukey's HSD test as a post-hoc test for a within-groups ANOVA test?

I understand Tukey's HSD is the most common post-hoc test for between-groups ANOVA. It appears some do not recommend or select it as a post-hoc test for within-groups ANOVA. Instead, they prefer ...
113 views

### Purpose of Scheffé's confidence intervals

Scheffé's method, when first written about by Henry Scheffé, was described as serving the purpose of identifying statistically significant contrasts in multiple-comparison's ANOVA problems. But here ...
95 views

### What do “single-step” and “multi-step” mean in post-hoc testing of ANOVAs?

From Wikipedia Post-hoc testing of ANOVAs Multiple comparison procedures are commonly used in an analysis of variance after obtaining a significant omnibus test result, like the ANOVA ...
37 views

### Is the likelihood ratio test for the global null considered to be an omnibus test

Continue from my last question about an omnibus test. When the (global) null is the intersection of several individual nulls, I was wondering if the likelihood ratio test for the global null is ...
28 views

### Questions about large-scale multiple testing

From Wikipedia Traditional methods for multiple comparisons adjustments focus on correcting for modest numbers of comparisons, often in an analysis of variance. A different set of techniques ...
215 views

### Relation between omnibus test and multiple comparison?

Wikipedia says Methods which rely on an omnibus test before proceeding to multiple comparisons. Typically these methods require a significant ANOVA/Tukey's range test before proceeding to ...
56 views

### The definitions of Simultaneous inference and selective inference?

What are the definitions of Simultaneous inference and selective inference? I think they are two different ways of doing multiple comparison. I encountered them from Wikipedia, which I think only ...
769 views

### Which test to use to compare proportions between 3 groups?

We are testing an e-mailing marketing campaign. On our initial test, we sent out two different e-mail types and had a third control group that did not receive an e-mail. Now we are getting back ...
72 views

### Joint test of change in variance across matched groups

I am monitoring variability in sentencing outcomes (custodial sentence length) amongst matched groups. The aim is to determine whether sentence variability went up or down after a policy was ...
108 views

### Multiple hypothesis tests, “comparing the expected values”

If I have two sets of random samples and I'd like to test whether there is some difference or not between them (or if one is better than the other) I'd test the null hypothesis $H_0: \mu_1 = \mu_2$ ...
64 views

### Multiple Comparisons help

I have already asked a similar question but did not explain my problem in entirety. Any help would be greatly appreciated! I am analyzing some data to which I think I need to make some kind of ...
39 views

### Multiple comparisons in economic game

I am analysing some data to which I think I need to make some kind of multiple comparisons correction. The data comes from an economic game (Dictator game) and is the form of donations made by ...
326 views

### Big Data vs multiple hypothesis testing?

Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics ...
51 views

### Group sequential Design

Can anyone recommend a paper which provides a nice introduction or summary of group sequential design.
35 views

### Is this a valid method to control the FWER?

I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
1k views

### How to calculate power (or sample size) for a multiple comparison experiment?

I collected data on 20 groups (with 30 elements each). A multiple comparison procedure (pairwise t-test with Holm correction) shows that in general there are three sets of groups: the high with 4 ...
93 views

### Number of tests for rare events and multiple comparisons problem?

I am conducting some statistical tests on a rare event (11 cases out of 171 patient sample). I have two main hypotheses, but in the mean time I also want to conduct some exploratory tests on 6 other ...
201 views

### Do I need P value Adjustment for 100 pairwise comparisons?

I made 100 comparisons by applying chi-square test (tajima relative rate test) and got 100 p values. Do I need P-value adjustment? If yes then which one correction will be suitable? I have explored ...
2k views

### How can you have a non-significant multiple regression model w/ significant predictors? [duplicate]

Possible Duplicate: Not-significant F but a significant coefficient in multiple linear regression How can a regression be significant yet all predictors be non-significant? Significance of ...
262 views

### On the applicability of Benjamini-Hochberg

If I've understood correctly, the Benjamini-Hochberg (BH) correction is used to correct for the rate of false discoveries (FDR) when testing a collection of $m$ random variables, $\{X_1, \ldots,X_m\}$ ...
180 views

### What test should I choose if I want to see if two groups are different from each other in many categories?

I am currently analysing data of a big webshop with over 31,000 vendors selling goods, i.e. each vendor is selling items over the webshop. The cool thing is that we know the gender of these vendors ...
185 views

### Repeated measurements with missing values

The data is measured in two biological replicates with two technical replicates, each. Each variable might be detected only in one or more replicates, in the others it is considered missing at ...
158 views

### What test is appropriate for this binned data?

I have collected questions from 10th grade students about genetic engineering before and after learning about the subject and classified the questions based on their cognitive level.. Before learning ...
533 views

### Practical significance, especially with percents: “standard” measure and threshold

I, like many people, dislike statistical significance testing. I would much rather measure "practical significance" / effect size. The problem is that I do not know of a "standard" way of doing so. ...
422 views

### On the corrections for multiple comparisons

I've used the Mann-Whitney U-test in order to check for possible differences between two relatively small groups (12 subject in each) on 15 different items. I didn't have any pre-defined assumption ...
255 views

### Why use Bonferroni approximation for experiment-wise alpha?

It seems the Bonferroni method (dividing experimentwise alpha by # of comparisons) for choosing the p level to fix the experimentwise alpha (when doing many pairwise comparisons) is more conservative ...
178 views

### Multiple FDR corrected experiments using the same data

I am testing many (500,000) genetic variants, and the tests are FDR corrected and give me a q-value. Normally I would just call everything with q < .05 significant. But in this case I am testing ...
794 views

### Experimentwise alpha for 2-way ANOVA

A two-way ANOVA yields 3 F ratios. Each F ratio has alpha of .05. Does that mean the experimentwise alpha level of the ANOVA is not .05 but rather is 1 - .95^3? If so, is there a recommended way to ...
10k views

### Significance of coefficients in linear regression: significant t-test vs non-significant F-statistic [duplicate]

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. R gives me the following p-values from the ...
205 views

### Sequential experiment: controlled design and metrics

This is a generic question. Let me put forth an example scenario. Say, I had 2 techniques for allocating my daily budget within 4 stocks. Upon allocation, I get the data about the stock's performance ...