Rejecting the null hypothesis when it is true.

learn more… | top users | synonyms

0
votes
0answers
36 views

Fairness of a Coin

A coin is tossed $40$ times. Define $T$ as the number of tails. i) Define the region of rejection by $|T-20|\geq5.$ Calculate $\alpha,$ the significance level --- $\displaystyle\alpha = P(y\leq15 ...
2
votes
1answer
37 views

How to estimate probability of $\geq$ n type I error in multiple testing comparison?

My question is about the calculation of the probability of making $\geq n$ type I error when $p$ independent statistical tests are made. I can calculate the probability of $\geq 1$ type I error with ...
3
votes
2answers
703 views

Is it a contradiction to do a hypothesis test on 1000 simulated datasets and never get a p value <0.05. Type 1 error question.

I am trying to calculate the type 1 error of a bootstrap hypothesis test procedure (won't go into the hypothesis test here). I ran the test on 1000 simulated datasets (simulated under the null ...
0
votes
1answer
50 views

Confidence interval and type 1 error cumulation

I have a sample of 15 subjects that did a test multiple times. Now I would like to see if the subjects differ in their mean performance. The usual approach would be a test for differences in means ...
0
votes
0answers
20 views

Type I errors on Hypothesis testing with KL divergence

I am performing a hypothesis test for data from an empirical distribution, $q$, where my null hypothesis is that the data is sampled from distribution $p_0$ and the alternative is that it is sampled ...
3
votes
2answers
67 views

Why is the complement to Power not $\alpha$?

i) Wrongly rejecting $H_0$ is called a type I error (controlled by $\alpha$). ii) Wrongly accepting $H_0$ is called a type II error (the probability of which is indicated by $\beta$). iii) Power is ...
0
votes
0answers
12 views

t-test : how to recover alpha when only have t-stat and Dof

In a t-test, is it possible to recover the value of alpha when you only have the degrees of freedom and the t-statistic? e.g. qt(0.05/2,100) = -1.983972 If I know that dof=100 and that ...
3
votes
0answers
30 views

Is the increase in Type I errors due to multi-sample t-tests overstated?

To illustrate, let's say you want to compare the means of 3 samples with t-tests. You would then have to run 3 tests to compare the means of each possible combination of samples. The explanations I ...
5
votes
1answer
102 views

How bad can heteroscedasticity be before causing problems?

I have two questions about heteroscedasticity in multiple regressions. According to my trusty textbook (Using Multivariate Statistics 2007, p.127), it says that deviations from ...
8
votes
3answers
258 views

Are probabilities of Type I and II errors negatively correlated?

In an elementary statistics class that I was a TA for, the professor stated that as the probability of a type I error $\alpha$ increases, the probability of a type II error $\beta$ decreases, and the ...
2
votes
0answers
49 views

Using simulation to estimate type-I error in Bayesian Tests

When doing a bayesian test, it is possible to estimate a "type-I error" of the test procedure by generating data from the null-hypothesis and running the bayesian test procedure several times. While ...
4
votes
2answers
80 views

Effects of blocking on type I and type II error rates

I am studying blocking in ANOVA and I am wondering about the following scenario. Suppose we did a Generalised randomised block design. Suppose SSBL = 0 and it also had not interaction effect with the ...
7
votes
3answers
416 views

Why is type I error not affected by different sample size - hypothesis testing? [duplicate]

I don't understand why the probability of getting a type I error when performing a hypothesis test, isn't affected. Increasing $n$ $\Rightarrow$ decreases standard deviation $\Rightarrow$ make the ...
8
votes
5answers
191 views

What is the relationship between $p$ values and Type I errors [duplicate]

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was ...
2
votes
0answers
35 views

Does N-1 correction for Pearson's Chi-Square apply to r x c tables generally?

Campbell (2007) recommended what he called the "N-1" correction for chi-square tests performed on two-by-two contingency tables. He described the correction as: "the K. Pearson chi-squared test ...
8
votes
1answer
241 views

Low sample size: LR vs F - test

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. Currently I am comparing negative binomial ...
0
votes
1answer
17 views

error summary metric

I am working on trying to get a summary metric that summarizes the results of four models (M1:M5) and preferably ranges from zero to one, with one being the best model and zero being the worst model. ...
0
votes
2answers
52 views

Simulating violations of regression assumptions

I'm wondering if anyone could provide some code (preferably in R) which demonstrates violated assumptions leading to type 1 errors. Some concrete examples of errors arising from assumption violations ...
9
votes
1answer
167 views

Increased Type I error - GLM

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. In my field of research (ecotoxicology) ...
28
votes
5answers
1k views

Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
5
votes
2answers
112 views

Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple ...
1
vote
1answer
28 views

Critical region for difference in proportions

A new virus breaks out on a cruise ship. I want to test the hypothesis that males and females are equally likely to contract the virus. I am going to test 100 men and 100 women. Presumably if I find ...
1
vote
2answers
187 views

Examples for Type I and Type II errors

I was checking on Type I (reject a true H$_{0}$) and Type II (fail to reject a false H$_{0}$) errors during hypothesis testing and got to to know the definitions. But I was looking for where and how ...
3
votes
2answers
272 views

Is p-value also the false discovery rate?

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) It states "For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery ...
0
votes
1answer
283 views

post-hoc test after logistic regression with interaction. Risk higher for type 1 or type 2 error?

I have a large data set (>1000 obs) and i'm performing regressions tests, both linear and logistic, on a series of clinical outcomes. In this test I verify the effect of interactions between two cat ...
5
votes
1answer
239 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
1
vote
1answer
46 views

Is it possible to randomly sample from single data set (Monte Carlo style) to create new data sets?

Background I understand Monte Carlo methods only superficially, but I understand you can repeatedly randomly sample, with or without replacement, from your data set to estimate population parameters ...
0
votes
0answers
27 views

Finding the critical region which minimizes the type 2 error

Kinda have another question about hypothesis testing. alpha is type 1 error while beta is type 2 error. I'm a bit confused in what I'm supposed to do. I attempted to answer the question but I ...
0
votes
0answers
23 views

Advice on how to analyze these data

In my experiment, I have 312 separate samples, each of which is made up of several different food items. I multiply each food by a specific multiplier to convert it to energy, and then sum the energy ...
1
vote
1answer
295 views

How to control for type-1-error inflation with multiple chi-squared tests?

I have two between-subject factors, each with two levels (so 4 conditions). Furthermore, I have one dependent variable (qualitative), also consisting of two levels. Now I want to make pairwise ...
0
votes
1answer
62 views

glm inflated error…why?

I'm pretty new to stats, so this may be dumb. I've been running a bunch of models on randomly generated data to try and develop my understanding of type 1 error. I've noticed that using ...
0
votes
1answer
76 views

Confidence Interval Coverage-error and Type I error

Could somebody explain to me the relationship between coverage error and type one errors in multiple comparisons testing, if there is one in fact? Does a coverage error occur when the true value of ...
1
vote
0answers
209 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...
2
votes
2answers
108 views

Comparing different conditions on a binomial distribution

I have some data where I have tested a binomial random variable under 4 conditions. The null hypothesis is that they all have equal means, alternative hypothesis is that one or more means differ from ...
1
vote
0answers
218 views

2x2x2 repeated measures anova: do I correct for type I error?

I have a 2x2x2 repeated measures design and am trying to decompose a 3-way interaction. As all my factors have 2 levels, I always reach the point that I am doing pairwise comparisons for the simple ...
6
votes
1answer
105 views

How to measure uncertainty of a parameter when false positives exist?

The main goal of my research is to measure the percentage of brown dwarf stars in the Pleiades star forming cluster that are actually double stars (i.e. the brown dwarf star has a companion brown ...
0
votes
0answers
162 views

Analyzing type I error rate in a simulation study using ANOVA-like methods

I've run a simulation study in order to determine type I error rate of a statistic.My simulation design includes threes factors as sample size (4 levels), test length or number of items (3 levels) and ...
1
vote
2answers
159 views

Three important issues

I don't understand three important issues : (1) Why do we calculate the test statistic under the null hypothesis ? (2) Why do we reject the null hypothesis when : p-value < significance level ? ...
0
votes
1answer
80 views

Can I make any tests statistically significant?

I am using a simple $\chi^2$ test to see if there is any impact of an independent variable (two charts) on the resulting choices (two choices). I found that as the number of sample increase, the ...
3
votes
2answers
352 views

Multiple resampling test/train dataset when choosing new models?

I have been reading several posts on testing multiple models on the same dataset, which can lead to problems controling type-1 errors. Mostly these posts have to do with data-mining on big datasets: ...
3
votes
1answer
542 views

Determining sample size with a proportion and binomial distribution

I am trying to learn some statistics using the book, Biometry by Sokal and Rohlf (3e). This is an exercise in the 5th chapter which covers probability, the binomial distribution, and Poisson ...
5
votes
1answer
294 views
1
vote
0answers
279 views

Calculating the size of Type 1 error, Type 2 error and power of the test

Let $X$ have a binomial distribution with parameter $n=5$ and $P\in [p:p=\frac{1}{4},\frac{1}{2}]$. The null hypothesis $H_{0}:P=\frac{1}{4}$ is rejected, and The alternative hypothesis ...
10
votes
1answer
3k views

How and when to use the Bonferroni adjustment

I have two questions regarding when to use a Bonferroni adjustment: Is it appropriate to use a Bonferroni adjustment in all cases of multiple testing? If one performs a test on a data set, then one ...
1
vote
1answer
288 views

Type 1 error Bayesian Stats

Is there anybody who can tell me or explain to me about finding the Type 1 error for a Bayesian clinical trial. Of a 100 patients I need 59 successful patients to exceed my posterior threshold. I ...
3
votes
2answers
1k views

Simple and paired t-tests and the control for multiple comparisons

Let's say that I have a group of subjects that were measured in two conditions (A and B) and I want to check three hypotheses: i) do data from A and B differ from each other?, ii) do data from A ...
3
votes
2answers
297 views

Rationale behind using ANOVA to compare means

Let's say we have 7 samples of data and we wish to test whether there is a significant difference between the means of each sample. We could use t-tests, which would require 21 different t-tests. ...
0
votes
0answers
52 views

What is the type 1 error of rejecting samples with low p-values?

Consider an amateur statistician who will discard any sample which does not lead him to reject his null hypothesis at an alpha=.05 significance level with a two-sided test. What is this statisticians ...
2
votes
1answer
2k views

Correcting for Type 1 error in multiple paired t-tests?

I'm wondering whether or not I should adjust the significance level of paired t-tests due to multiple tests (to avoid the possibility of Type 1 error), although the tests are independent. Here's what ...
3
votes
1answer
310 views

On FDA guidance about Bayesian practice

US FDA authorizes the use of Bayesian statistics with informative priors (in certain contexts): ...