Rejecting the null hypothesis when it is true.

learn more… | top users | synonyms

0
votes
1answer
35 views

glm inflated error…why?

I'm pretty new to stats, so this may be dumb. I've been running a bunch of models on randomly generated data to try and develop my understanding of type 1 error. I've noticed that using ...
0
votes
1answer
17 views

Confidence Interval Coverage-error and Type I error

Could somebody explain to me the relationship between coverage error and type one errors in multiple comparisons testing, if there is one in fact? Does a coverage error occur when the true value of ...
1
vote
0answers
13 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...
0
votes
0answers
11 views

Identifying Type 1 errors using confidence intervals

I'm working with an SPSS data set, a Monte Carlo study with 5000 trials. There are 5 means and therefore 10 comparisons for each trial. I'm provided with the t-test significance of each comparison ...
2
votes
2answers
41 views

Comparing different conditions on a binomial distribution

I have some data where I have tested a binomial random variable under 4 conditions. The null hypothesis is that they all have equal means, alternative hypothesis is that one or more means differ from ...
1
vote
0answers
40 views

2x2x2 repeated measures anova: do I correct for type I error?

I have a 2x2x2 repeated measures design and am trying to decompose a 3-way interaction. As all my factors have 2 levels, I always reach the point that I am doing pairwise comparisons for the simple ...
6
votes
1answer
99 views

How to measure uncertainty of a parameter when false positives exist?

The main goal of my research is to measure the percentage of brown dwarf stars in the Pleiades star forming cluster that are actually double stars (i.e. the brown dwarf star has a companion brown ...
0
votes
0answers
52 views

Analyzing type I error rate in a simulation study using ANOVA-like methods

I've run a simulation study in order to determine type I error rate of a statistic.My simulation design includes threes factors as sample size (4 levels), test length or number of items (3 levels) and ...
1
vote
2answers
127 views

Three important issues

I don't understand three important issues : (1) Why do we calculate the test statistic under the null hypothesis ? (2) Why do we reject the null hypothesis when : p-value < significance level ? ...
0
votes
1answer
64 views

Can I make any tests statistically significant?

I am using a simple $\chi^2$ test to see if there is any impact of an independent variable (two charts) on the resulting choices (two choices). I found that as the number of sample increase, the ...
2
votes
2answers
99 views

Multiple resampling test/train dataset when choosing new models?

I have been reading several posts on testing multiple models on the same dataset, which can lead to problems controling type-1 errors. Mostly these posts have to do with data-mining on big datasets: ...
3
votes
1answer
128 views

Determining sample size with a proportion and binomial distribution

I am trying to learn some statistics using the book, Biometry by Sokal and Rohlf (3e). This is an exercise in the 5th chapter which covers probability, the binomial distribution, and Poisson ...
5
votes
1answer
118 views
1
vote
0answers
110 views

Calculating the size of Type 1 error, Type 2 error and power of the test

Let $X$ have a binomial distribution with parameter $n=5$ and $P\in [p:p=\frac{1}{4},\frac{1}{2}]$. The null hypothesis $H_{0}:P=\frac{1}{4}$ is rejected, and The alternative hypothesis ...
0
votes
0answers
22 views

Checking significance of all prefixes of a coin flip sequence

Derived from this blog post and similar to this question: Given a random walk generated from flipping a coin, what is the function for the envelope that a fair coin will not exit at any point (with ...
10
votes
1answer
317 views

How and when to use the Bonferroni adjustment

I have two questions regarding when to use a Bonferroni adjustment: Is it appropriate to use a Bonferroni adjustment in all cases of multiple testing? If one performs a test on a data set, then one ...
0
votes
0answers
31 views

Type 1 error for fixed Bayesian trial

this is my R code for determining the number of successful patients needed for a clinical trial. I am trying to determine the type 1 error for the trial, can somebody give any advice or help. ...
0
votes
0answers
43 views

Adaptive Bayesian design for power calculation (type 1 error)

Can anybody give me some advice on how to incorporate power calculations into my code. I'm trying show that the operating characteristics of this adaptive Bayesian design for assumed probabilities ...
1
vote
1answer
129 views

Type 1 error Bayesian Stats

Is there anybody who can tell me or explain to me about finding the Type 1 error for a Bayesian clinical trial. Of a 100 patients I need 59 successful patients to exceed my posterior threshold. I ...
3
votes
2answers
676 views

Simple and paired t-tests and the control for multiple comparisons

Let's say that I have a group of subjects that were measured in two conditions (A and B) and I want to check three hypotheses: i) do data from A and B differ from each other?, ii) do data from A ...
3
votes
2answers
181 views

Rationale behind using ANOVA to compare means

Let's say we have 7 samples of data and we wish to test whether there is a significant difference between the means of each sample. We could use t-tests, which would require 21 different t-tests. ...
0
votes
0answers
42 views

What is the type 1 error of rejecting samples with low p-values?

Consider an amateur statistician who will discard any sample which does not lead him to reject his null hypothesis at an alpha=.05 significance level with a two-sided test. What is this statisticians ...
2
votes
1answer
484 views

Correcting for Type 1 error in multiple paired t-tests?

I'm wondering whether or not I should adjust the significance level of paired t-tests due to multiple tests (to avoid the possibility of Type 1 error), although the tests are independent. Here's what ...
3
votes
1answer
183 views

On FDA guidance about Bayesian practice

US FDA authorizes the use of Bayesian statistics with informative priors (in certain contexts): ...
2
votes
1answer
124 views

Is a MANOVA justified?

I have to analyze the data from a study in which two groups of subjects (total sample = 30) were tested on 4 cognitive tasks, all realtive to executive functions. For each task, I have accuracy score ...
1
vote
1answer
230 views

Determining number of tests to correct for using Bonferroni

I am doing final revisions to my dissertation, which includes a large number of regressions. I have six outcome variables (DVs) and I am testing the effect of four primary (correlated) IVs, but I am ...
13
votes
2answers
301 views

Optional stopping rules not in textbooks

Stopping rules affect the relationship between P-values and the error rates associated with decisions. A recent paper by Simmons et al. 2011 coins the term researcher degrees of freedom to describe a ...
2
votes
1answer
138 views

Controlling type I error in permutation tests

I'm implementing a permutation test for a test statistic with unknown distribution. Apart from my concrete situation (see below), I'd like to know first If and under what circumstances permutation ...
2
votes
2answers
60 views

Conditions under which the power curve is not smooth or continuous in Hypothesis testing

Not sure if this is a valid question. I was wondering if there is are good answers to this question: Under what conditions is the power curve not smooth or continuous in Hypothesis testing? ...
4
votes
3answers
527 views

Experiment or simulation to undestand type I and type II errors

I am studying Type I and Type II errors and basic concepts of testing hypotheses. To better develop my intuition I would like to write a few simple simulations. What I'm hoping is to get a base ...
1
vote
1answer
107 views

Prove this theorem related with specification tests

I'm not a mathematician and I need to prove what this theorem says. I think is easy and I know how it works, but definitively I'm not too much rigorous to make a demonstration. Can anybody help me? I ...
2
votes
1answer
232 views

The impact of non-normality (contaminated normal) on type 1 error and power when sigma is known

When sigma is unknown, contaminated normal distributions tend to have (1) the issue of the actual Type 1 error rate being lower than the nominal level and (2) the issue of reduced power. This pattern ...
3
votes
1answer
180 views

Overall type I error with dependent tests

Imagine we test 2 dependent samples: For example, we first test an overall sample (treatment vs. control) and then a subset of this overall sample (treatment sbg vs. control sbg). In this case, we are ...
2
votes
1answer
359 views

Frequentist properties of p-values in relation to type I error

This question is related to frequentist properties of p-values and their relation to type I error and why the results from an online simulation differ from what I would have expected. Assume that I ...
1
vote
0answers
53 views

Relativization and tackling an overdispersed dataset

If there is a species that is hyperabundant in a dataset and makes up 284/1638 individuals yet originated from just 1 of 10 traps. Is a quassi-poisson GLM sufficient providing residual deviance is ...
0
votes
1answer
478 views

What type of error correction do I have to use after Wilcoxon matched pairs test?

Do I have to use Bonferroni's or Holm's corrections? Please see this previous question for my example data and the design. (Shortly, I have three or more dependent groups.)
6
votes
2answers
2k views

How to test whether a regression coefficient is moderated by a grouping variable?

I have a regression done on two groups of the sample based on a moderating variable (say gender). I'm doing a simple test for the moderating effect by checking whether the significance of the ...
11
votes
1answer
650 views

Overall type I error when repeatedly testing accumulating data

I have a question about group sequential methods. According to Wikipedia: In a randomized trial with two treatment groups, classical group sequential testing is used in the following manner: If n ...
37
votes
24answers
10k views

Is there a way to remember the definitions of Type I and Type II Errors?

I'm not a statistician by education, I'm a software engineer. Yet statistics comes up a lot. In fact, questions specifically about Type I and Type II error are coming up a lot in the course of my ...