Rejecting the null hypothesis when it is true.

learn more… | top users | synonyms

2
votes
0answers
20 views

Does N-1 correction for Pearson's Chi-Square apply to r x c tables generally?

Campbell (2007) recommended what he called the "N-1" correction for chi-square tests performed on two-by-two contingency tables. He described the correction as: "the K. Pearson chi-squared test ...
8
votes
1answer
147 views

Low sample size: LR vs F - test

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. Currently I am comparing negative binomial ...
0
votes
1answer
12 views

error summary metric

I am working on trying to get a summary metric that summarizes the results of four models (M1:M5) and preferably ranges from zero to one, with one being the best model and zero being the worst model. ...
0
votes
2answers
34 views

Simulating violations of regression assumptions

I'm wondering if anyone could provide some code (preferably in R) which demonstrates violated assumptions leading to type 1 errors. Some concrete examples of errors arising from assumption violations ...
9
votes
1answer
131 views

Increased Type I error - GLM

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. In my field of research (ecotoxicology) ...
24
votes
4answers
843 views

Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
5
votes
2answers
101 views

Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple ...
1
vote
1answer
27 views

Critical region for difference in proportions

A new virus breaks out on a cruise ship. I want to test the hypothesis that males and females are equally likely to contract the virus. I am going to test 100 men and 100 women. Presumably if I find ...
1
vote
2answers
110 views

Examples for Type I and Type II errors

I was checking on Type I (reject a true H$_{0}$) and Type II (fail to reject a false H$_{0}$) errors during hypothesis testing and got to to know the definitions. But I was looking for where and how ...
3
votes
2answers
216 views

Is p-value also the false discovery rate?

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) It states "For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery ...
0
votes
1answer
85 views

post-hoc test after logistic regression with interaction. Risk higher for type 1 or type 2 error?

I have a large data set (>1000 obs) and i'm performing regressions tests, both linear and logistic, on a series of clinical outcomes. In this test I verify the effect of interactions between two cat ...
5
votes
1answer
196 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
0
votes
1answer
31 views

Is it possible to randomly sample from single data set (Monte Carlo style) to create new data sets?

Background I understand Monte Carlo methods only superficially, but I understand you can repeatedly randomly sample, with or without replacement, from your data set to estimate population parameters ...
0
votes
0answers
21 views

Finding the critical region which minimizes the type 2 error

Kinda have another question about hypothesis testing. alpha is type 1 error while beta is type 2 error. I'm a bit confused in what I'm supposed to do. I attempted to answer the question but I ...
0
votes
0answers
19 views

Advice on how to analyze these data

In my experiment, I have 312 separate samples, each of which is made up of several different food items. I multiply each food by a specific multiplier to convert it to energy, and then sum the energy ...
1
vote
1answer
134 views

How to control for type-1-error inflation with multiple chi-squared tests?

I have two between-subject factors, each with two levels (so 4 conditions). Furthermore, I have one dependent variable (qualitative), also consisting of two levels. Now I want to make pairwise ...
0
votes
1answer
53 views

glm inflated error…why?

I'm pretty new to stats, so this may be dumb. I've been running a bunch of models on randomly generated data to try and develop my understanding of type 1 error. I've noticed that using ...
0
votes
1answer
45 views

Confidence Interval Coverage-error and Type I error

Could somebody explain to me the relationship between coverage error and type one errors in multiple comparisons testing, if there is one in fact? Does a coverage error occur when the true value of ...
1
vote
0answers
159 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...
0
votes
0answers
20 views

Identifying Type 1 errors using confidence intervals

I'm working with an SPSS data set, a Monte Carlo study with 5000 trials. There are 5 means and therefore 10 comparisons for each trial. I'm provided with the t-test significance of each comparison ...
2
votes
2answers
86 views

Comparing different conditions on a binomial distribution

I have some data where I have tested a binomial random variable under 4 conditions. The null hypothesis is that they all have equal means, alternative hypothesis is that one or more means differ from ...
1
vote
0answers
164 views

2x2x2 repeated measures anova: do I correct for type I error?

I have a 2x2x2 repeated measures design and am trying to decompose a 3-way interaction. As all my factors have 2 levels, I always reach the point that I am doing pairwise comparisons for the simple ...
6
votes
1answer
103 views

How to measure uncertainty of a parameter when false positives exist?

The main goal of my research is to measure the percentage of brown dwarf stars in the Pleiades star forming cluster that are actually double stars (i.e. the brown dwarf star has a companion brown ...
0
votes
0answers
124 views

Analyzing type I error rate in a simulation study using ANOVA-like methods

I've run a simulation study in order to determine type I error rate of a statistic.My simulation design includes threes factors as sample size (4 levels), test length or number of items (3 levels) and ...
1
vote
2answers
143 views

Three important issues

I don't understand three important issues : (1) Why do we calculate the test statistic under the null hypothesis ? (2) Why do we reject the null hypothesis when : p-value < significance level ? ...
0
votes
1answer
72 views

Can I make any tests statistically significant?

I am using a simple $\chi^2$ test to see if there is any impact of an independent variable (two charts) on the resulting choices (two choices). I found that as the number of sample increase, the ...
2
votes
2answers
193 views

Multiple resampling test/train dataset when choosing new models?

I have been reading several posts on testing multiple models on the same dataset, which can lead to problems controling type-1 errors. Mostly these posts have to do with data-mining on big datasets: ...
3
votes
1answer
442 views

Determining sample size with a proportion and binomial distribution

I am trying to learn some statistics using the book, Biometry by Sokal and Rohlf (3e). This is an exercise in the 5th chapter which covers probability, the binomial distribution, and Poisson ...
5
votes
1answer
217 views
1
vote
0answers
219 views

Calculating the size of Type 1 error, Type 2 error and power of the test

Let $X$ have a binomial distribution with parameter $n=5$ and $P\in [p:p=\frac{1}{4},\frac{1}{2}]$. The null hypothesis $H_{0}:P=\frac{1}{4}$ is rejected, and The alternative hypothesis ...
10
votes
1answer
1k views

How and when to use the Bonferroni adjustment

I have two questions regarding when to use a Bonferroni adjustment: Is it appropriate to use a Bonferroni adjustment in all cases of multiple testing? If one performs a test on a data set, then one ...
1
vote
1answer
222 views

Type 1 error Bayesian Stats

Is there anybody who can tell me or explain to me about finding the Type 1 error for a Bayesian clinical trial. Of a 100 patients I need 59 successful patients to exceed my posterior threshold. I ...
3
votes
2answers
1k views

Simple and paired t-tests and the control for multiple comparisons

Let's say that I have a group of subjects that were measured in two conditions (A and B) and I want to check three hypotheses: i) do data from A and B differ from each other?, ii) do data from A ...
3
votes
2answers
240 views

Rationale behind using ANOVA to compare means

Let's say we have 7 samples of data and we wish to test whether there is a significant difference between the means of each sample. We could use t-tests, which would require 21 different t-tests. ...
0
votes
0answers
47 views

What is the type 1 error of rejecting samples with low p-values?

Consider an amateur statistician who will discard any sample which does not lead him to reject his null hypothesis at an alpha=.05 significance level with a two-sided test. What is this statisticians ...
2
votes
1answer
1k views

Correcting for Type 1 error in multiple paired t-tests?

I'm wondering whether or not I should adjust the significance level of paired t-tests due to multiple tests (to avoid the possibility of Type 1 error), although the tests are independent. Here's what ...
3
votes
1answer
258 views

On FDA guidance about Bayesian practice

US FDA authorizes the use of Bayesian statistics with informative priors (in certain contexts): ...
3
votes
1answer
276 views

Is a MANOVA justified?

I have to analyze the data from a study in which two groups of subjects (total sample = 30) were tested on 4 cognitive tasks, all realtive to executive functions. For each task, I have accuracy score ...
1
vote
1answer
318 views

Determining number of tests to correct for using Bonferroni

I am doing final revisions to my dissertation, which includes a large number of regressions. I have six outcome variables (DVs) and I am testing the effect of four primary (correlated) IVs, but I am ...
13
votes
2answers
343 views

Optional stopping rules not in textbooks

Stopping rules affect the relationship between P-values and the error rates associated with decisions. A recent paper by Simmons et al. 2011 coins the term researcher degrees of freedom to describe a ...
2
votes
1answer
176 views

Controlling type I error in permutation tests

I'm implementing a permutation test for a test statistic with unknown distribution. Apart from my concrete situation (see below), I'd like to know first If and under what circumstances permutation ...
2
votes
2answers
69 views

Conditions under which the power curve is not smooth or continuous in Hypothesis testing

Not sure if this is a valid question. I was wondering if there is are good answers to this question: Under what conditions is the power curve not smooth or continuous in Hypothesis testing? ...
4
votes
3answers
765 views

Experiment or simulation to undestand type I and type II errors

I am studying Type I and Type II errors and basic concepts of testing hypotheses. To better develop my intuition I would like to write a few simple simulations. What I'm hoping is to get a base ...
1
vote
1answer
146 views

Prove this theorem related with specification tests

I'm not a mathematician and I need to prove what this theorem says. I think is easy and I know how it works, but definitively I'm not too much rigorous to make a demonstration. Can anybody help me? I ...
2
votes
1answer
304 views

The impact of non-normality (contaminated normal) on type 1 error and power when sigma is known

When sigma is unknown, contaminated normal distributions tend to have (1) the issue of the actual Type 1 error rate being lower than the nominal level and (2) the issue of reduced power. This pattern ...
3
votes
1answer
225 views

Overall type I error with dependent tests

Imagine we test 2 dependent samples: For example, we first test an overall sample (treatment vs. control) and then a subset of this overall sample (treatment sbg vs. control sbg). In this case, we are ...
3
votes
1answer
458 views

Frequentist properties of p-values in relation to type I error

This question is related to frequentist properties of p-values and their relation to type I error and why the results from an online simulation differ from what I would have expected. Assume that I ...
1
vote
0answers
59 views

Relativization and tackling an overdispersed dataset

If there is a species that is hyperabundant in a dataset and makes up 284/1638 individuals yet originated from just 1 of 10 traps. Is a quassi-poisson GLM sufficient providing residual deviance is ...
0
votes
1answer
564 views

What type of error correction do I have to use after Wilcoxon matched pairs test?

Do I have to use Bonferroni's or Holm's corrections? Please see this previous question for my example data and the design. (Shortly, I have three or more dependent groups.)
6
votes
2answers
3k views

How to test whether a regression coefficient is moderated by a grouping variable?

I have a regression done on two groups of the sample based on a moderating variable (say gender). I'm doing a simple test for the moderating effect by checking whether the significance of the ...