Tagged Questions

1
vote
1answer
22 views

Relation between omnibus test and multiple comparison?

Wikipedia says Methods which rely on an omnibus test before proceeding to multiple comparisons. Typically these methods require a significant ANOVA/Tukey's range test before proceeding to ...
0
votes
0answers
11 views

The definitions of Simultaneous inference and selective inference?

What are the definitions of Simultaneous inference and selective inference? I think they are two different ways of doing multiple comparison. I encountered them from Wikipedia, which I think only ...
2
votes
2answers
144 views

Which test to use to compare proportions between 3 groups?

We are testing an e-mailing marketing campaign. On our initial test, we sent out two different e-mail types and had a third control group that did not receive an e-mail. Now we are getting back ...
2
votes
1answer
44 views

Joint test of change in variance across matched groups

I am monitoring variability in sentencing outcomes (custodial sentence length) amongst matched groups. The aim is to determine whether sentence variability went up or down after a policy was ...
0
votes
1answer
48 views

Multiple hypothesis tests, “comparing the expected values”

If I have two sets of random samples and I'd like to test whether there is some difference or not between them (or if one is better than the other) I'd test the null hypothesis $H_0: \mu_1 = \mu_2$ ...
0
votes
1answer
52 views

Multiple Comparisons help

I have already asked a similar question but did not explain my problem in entirety. Any help would be greatly appreciated! I am analyzing some data to which I think I need to make some kind of ...
1
vote
0answers
37 views

Multiple comparisons in economic game

I am analysing some data to which I think I need to make some kind of multiple comparisons correction. The data comes from an economic game (Dictator game) and is the form of donations made by ...
4
votes
3answers
216 views

Big Data vs multiple hypothesis testing?

Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics ...
0
votes
1answer
39 views

Group sequential Design

Can anyone recommend a paper which provides a nice introduction or summary of group sequential design.
2
votes
0answers
32 views

Is this a valid method to control the FWER?

I have a huge number, say $M$, of hypotheses that are potentially correlated. I have a dataset $D$ of random samples from an unknown distribution and I want to do test the hypotheses for significance ...
1
vote
0answers
70 views

Number of tests for rare events and multiple comparisons problem?

I am conducting some statistical tests on a rare event (11 cases out of 171 patient sample). I have two main hypotheses, but in the mean time I also want to conduct some exploratory tests on 6 other ...
1
vote
2answers
174 views

Do I need P value Adjustment for 100 pairwise comparisons?

I made 100 comparisons by applying chi-square test (tajima relative rate test) and got 100 p values. Do I need P-value adjustment? If yes then which one correction will be suitable? I have explored ...
0
votes
2answers
1k views

How can you have a non-significant multiple regression model w/ significant predictors? [duplicate]

Possible Duplicate: Not-significant F but a significant coefficient in multiple linear regression How can a regression be significant yet all predictors be non-significant? Significance of ...
5
votes
2answers
228 views

On the applicability of Benjamini-Hochberg

If I've understood correctly, the Benjamini-Hochberg (BH) correction is used to correct for the rate of false discoveries (FDR) when testing a collection of $m$ random variables, $\{X_1, \ldots,X_m\}$ ...
5
votes
1answer
165 views

What test should I choose if I want to see if two groups are different from each other in many categories?

I am currently analysing data of a big webshop with over 31,000 vendors selling goods, i.e. each vendor is selling items over the webshop. The cool thing is that we know the gender of these vendors ...
2
votes
1answer
141 views

Repeated measurements with missing values

The data is measured in two biological replicates with two technical replicates, each. Each variable might be detected only in one or more replicates, in the others it is considered missing at ...
2
votes
2answers
140 views

What test is appropriate for this binned data?

I have collected questions from 10th grade students about genetic engineering before and after learning about the subject and classified the questions based on their cognitive level.. Before learning ...
4
votes
4answers
327 views

Practical significance, especially with percents: “standard” measure and threshold

I, like many people, dislike statistical significance testing. I would much rather measure "practical significance" / effect size. The problem is that I do not know of a "standard" way of doing so. ...
3
votes
2answers
294 views

On the corrections for multiple comparisons

I've used the Mann-Whitney U-test in order to check for possible differences between two relatively small groups (12 subject in each) on 15 different items. I didn't have any pre-defined assumption ...
2
votes
2answers
224 views

Why use Bonferroni approximation for experiment-wise alpha?

It seems the Bonferroni method (dividing experimentwise alpha by # of comparisons) for choosing the p level to fix the experimentwise alpha (when doing many pairwise comparisons) is more conservative ...
6
votes
1answer
131 views

Multiple FDR corrected experiments using the same data

I am testing many (500,000) genetic variants, and the tests are FDR corrected and give me a q-value. Normally I would just call everything with q < .05 significant. But in this case I am testing ...
3
votes
2answers
537 views

Experimentwise alpha for 2-way ANOVA

A two-way ANOVA yields 3 F ratios. Each F ratio has alpha of .05. Does that mean the experimentwise alpha level of the ANOVA is not .05 but rather is 1 - .95^3? If so, is there a recommended way to ...
8
votes
3answers
8k views

Significance of coefficients in linear regression: significant t-test vs non-significant F-statistic [duplicate]

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. R gives me the following p-values from the ...
1
vote
1answer
178 views

Sequential experiment: controlled design and metrics

This is a generic question. Let me put forth an example scenario. Say, I had 2 techniques for allocating my daily budget within 4 stocks. Upon allocation, I get the data about the stock's performance ...
5
votes
3answers
1k views

Alpha adjustment for multiple testing

I understand the logic of alpha adjustment for multiple testing. However, I am confused as to whether this correction should be applied to all tests on a dataset or only the pairwise comparison in ...
3
votes
2answers
235 views

What might be a clear, practical definition for a “family of hypotheses” (with respect to familywise error rate)?

When trying to evaluate what constitutes a family of hypotheses within an experiment/project/analysis, I've found "similar in purpose" and "similar in content" given as guidelines for delimiting ...
4
votes
1answer
98 views

Mean of one group vs. the rest of the sample

I have data from a survey with Likert-type questions (Rate how strongly you feel about this issue on a scale from 1 to 10). We know the respondents home town (as well as some other information, such ...
5
votes
2answers
469 views

Understanding MANOVA in case of a single predictor

I'm trying to understand the statistical analysis I saw in a clinical study. They measured performance of 3 groups of subjects with a series of performance measures (A, B, C, ..., N). The objective ...
4
votes
2answers
212 views

How to test for association between proportions with multiple variables?

I have tested 100 bacterial isolates for resistance to 6 antibiotics (tetracycline, erythromycin, kanamycin, streptomycin, gentamicin, chloramphenicol). The result for any antibiotic could be either ...
0
votes
2answers
121 views

Probability of accidentally producing significant results

I have a problem, which I think should be very common, but I mostly have been taught stochastic at university and most of my statistics knowledge is self-taught. Hence I have no clue on how to look ...
1
vote
0answers
36 views

Metric for cross-comparing vectors of multiple values for a few class labels

There are 20 individuals who get tested for a particular task on 3 consecutive days. They are considered to be iid. These results are stored in a 60 element vector. There is a similar test on 3 ...
1
vote
1answer
87 views

Identifying largest proportion in a set based on a common set of companies

To run a certain activity, a population of companies use a finite set of inputs. There are 8 different inputs, but each company may not use all of them. For instance, company 1 uses inputs A, B and ...
10
votes
2answers
444 views

Has anyone solved PTLOS exercise 4.1?

This an exercise given in Probability Theory: The Logic of Science by Edwin Jaynes, 2003. There is a partial solution here. I have worked out a more general partial solution, and was wondering if ...
0
votes
2answers
934 views

Analysing questionnaire data

I want to test if there is a rivalry among two siblings in a family. I have 15 questions in my study and I let my 100 respondents ( distributed equally to two siblings) ranked them 1 to 15. How ...
2
votes
1answer
306 views

Multiple comparisons on nested subsets of data

Suppose I am doing some experimental procedure on two treatment groups. The procedure has several stages, each of which may fail. Failure at any stage halts the experiment. If all stages are passed ...
1
vote
1answer
145 views

Removing macro-level time variance

The title might be a bit misleading. Unfortunately statistics is not my area of forte, so gentle guidance along the right path is much appreciated. That said, here's my problem: I'm working on ...
20
votes
4answers
2k views

Why is multiple comparison a problem?

I find it hard to understand what really is the issue with multiple comparisons. With a simple analogy, it is said that a person who will make many decisions will make many mistakes. So very ...