Rejecting the null hypothesis when it is true.

learn more… | top users | synonyms

4
votes
3answers
97 views

Does false discovery rate depend on the p-value or only on the alpha level?

Let's say I get a p-value of 0.001. I know that alpha level dictates the probability of a type I error, so if I get a result this significant, is my false discovery rate (FDR) lower than if I were to ...
1
vote
1answer
31 views

Dependence, independence, multiple testing, and alpha error correction

although I have studied psychology and feel I should know a lot more about issues such as this one, I have a quite basic question (or maybe not so basic question, considering all the fuss about this) ...
1
vote
0answers
33 views

Correcting for family-wise error rate with series of repeated measures ANOVA?

I am trying to make requested revisions to an accepted manuscript, and I am baffled by the following comment from a reviewer: "Eight hypothesis tests are reported in the final paragraph of the ...
0
votes
0answers
17 views

Using Likelihood Ratio Test to deal with heteroscedastic data results in unreliable results

Suppose $Y$ and $x$ are not related. Therefore the linear regression analysis should not reject the null hypothesis ($H_0: b=0$) in $E(Y) = a+bx$. Suppose the variance in $Y$ increase with $x$ (i.e., ...
3
votes
1answer
78 views

What is the name for this unintuitive result with OLS on a very “asymmetric” regressand, and how should it be addressed?

Say our sample consists of about a hundred Belgian (x = 0) and Swiss (x = 1) chocolate bars. We test them to see if they have safe (y = 1) or lethal (y = 0) levels of arsenic. As it turn out, 90% are ...
0
votes
1answer
46 views

Fairness of a Coin

A coin is tossed $40$ times. Define $T$ as the number of tails. i) Define the region of rejection by $|T-20|\geq5.$ Calculate $\alpha,$ the significance level --- $\displaystyle\alpha = P(y\leq15 ...
2
votes
1answer
46 views

How to estimate probability of $\geq$ n type I error in multiple testing comparison?

My question is about the calculation of the probability of making $\geq n$ type I error when $p$ independent statistical tests are made. I can calculate the probability of $\geq 1$ type I error with ...
3
votes
2answers
730 views

Is it a contradiction to do a hypothesis test on 1000 simulated datasets and never get a p value <0.05. Type 1 error question.

I am trying to calculate the type 1 error of a bootstrap hypothesis test procedure (won't go into the hypothesis test here). I ran the test on 1000 simulated datasets (simulated under the null ...
0
votes
1answer
83 views

Confidence interval and type 1 error cumulation

I have a sample of 15 subjects that did a test multiple times. Now I would like to see if the subjects differ in their mean performance. The usual approach would be a test for differences in means ...
0
votes
0answers
33 views

Type I errors on Hypothesis testing with KL divergence

I am performing a hypothesis test for data from an empirical distribution, $q$, where my null hypothesis is that the data is sampled from distribution $p_0$ and the alternative is that it is sampled ...
4
votes
2answers
93 views

Why is the complement to Power not $\alpha$?

i) Wrongly rejecting $H_0$ is called a type I error (controlled by $\alpha$). ii) Wrongly accepting $H_0$ is called a type II error (the probability of which is indicated by $\beta$). iii) Power is ...
0
votes
0answers
14 views

t-test : how to recover alpha when only have t-stat and Dof

In a t-test, is it possible to recover the value of alpha when you only have the degrees of freedom and the t-statistic? e.g. qt(0.05/2,100) = -1.983972 If I know that dof=100 and that ...
3
votes
0answers
37 views

Is the increase in Type I errors due to multi-sample t-tests overstated?

To illustrate, let's say you want to compare the means of 3 samples with t-tests. You would then have to run 3 tests to compare the means of each possible combination of samples. The explanations I ...
5
votes
1answer
163 views

How bad can heteroscedasticity be before causing problems?

I have two questions about heteroscedasticity in multiple regressions. According to my trusty textbook (Using Multivariate Statistics 2007, p.127), it says that deviations from ...
8
votes
3answers
290 views

Are probabilities of Type I and II errors negatively correlated?

In an elementary statistics class that I was a TA for, the professor stated that as the probability of a type I error $\alpha$ increases, the probability of a type II error $\beta$ decreases, and the ...
2
votes
0answers
60 views

Using simulation to estimate type-I error in Bayesian Tests

When doing a bayesian test, it is possible to estimate a "type-I error" of the test procedure by generating data from the null-hypothesis and running the bayesian test procedure several times. While ...
4
votes
2answers
117 views

Effects of blocking on type I and type II error rates

I am studying blocking in ANOVA and I am wondering about the following scenario. Suppose we did a Generalised randomised block design. Suppose SSBL = 0 and it also had not interaction effect with the ...
7
votes
3answers
452 views

Why is type I error not affected by different sample size - hypothesis testing? [duplicate]

I don't understand why the probability of getting a type I error when performing a hypothesis test, isn't affected. Increasing $n$ $\Rightarrow$ decreases standard deviation $\Rightarrow$ make the ...
8
votes
5answers
234 views

What is the relationship between $p$ values and Type I errors [duplicate]

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was ...
2
votes
0answers
106 views

rejection region for correlated bivariate normal

STATEMENT OF PROBLEM: Suppose $ \left( \begin{array}{ccc} \ Z_1 \\ Z_2 \end{array} \right)$ follows a Bivariate standard normal with covariance $ \rho $ $ \left( \begin{array}{ccc} \ Z_1 \\ Z_2 ...
2
votes
0answers
50 views

Does N-1 correction for Pearson's Chi-Square apply to r x c tables generally?

Campbell (2007) recommended what he called the "N-1" correction for chi-square tests performed on two-by-two contingency tables. He described the correction as: "the K. Pearson chi-squared test ...
8
votes
1answer
301 views

Low sample size: LR vs F - test

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. Currently I am comparing negative binomial ...
1
vote
3answers
155 views

What is the basis of setting critical p-value value in stepwise regression?

In statistical software like MINITAB and SAS, the default alpha value (critical p-value) is set as 0.15. I would like to know (1) if there is any statistic basis to set it as 0.15 and (2) if this is a ...
0
votes
1answer
20 views

error summary metric

I am working on trying to get a summary metric that summarizes the results of four models (M1:M5) and preferably ranges from zero to one, with one being the best model and zero being the worst model. ...
0
votes
2answers
68 views

Simulating violations of regression assumptions

I'm wondering if anyone could provide some code (preferably in R) which demonstrates violated assumptions leading to type 1 errors. Some concrete examples of errors arising from assumption violations ...
9
votes
1answer
186 views

Increased Type I error - GLM

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. In my field of research (ecotoxicology) ...
32
votes
5answers
1k views

Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
5
votes
2answers
121 views

Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple ...
1
vote
1answer
29 views

Critical region for difference in proportions

A new virus breaks out on a cruise ship. I want to test the hypothesis that males and females are equally likely to contract the virus. I am going to test 100 men and 100 women. Presumably if I find ...
1
vote
2answers
253 views

Examples for Type I and Type II errors

I was checking on Type I (reject a true H$_{0}$) and Type II (fail to reject a false H$_{0}$) errors during hypothesis testing and got to to know the definitions. But I was looking for where and how ...
3
votes
2answers
406 views

Is p-value also the false discovery rate?

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) It states "For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery ...
0
votes
1answer
428 views

post-hoc test after logistic regression with interaction. Risk higher for type 1 or type 2 error?

I have a large data set (>1000 obs) and i'm performing regressions tests, both linear and logistic, on a series of clinical outcomes. In this test I verify the effect of interactions between two cat ...
6
votes
2answers
284 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
1
vote
1answer
61 views

Is it possible to randomly sample from single data set (Monte Carlo style) to create new data sets?

Background I understand Monte Carlo methods only superficially, but I understand you can repeatedly randomly sample, with or without replacement, from your data set to estimate population parameters ...
1
vote
1answer
429 views

How to control for type-1-error inflation with multiple chi-squared tests?

I have two between-subject factors, each with two levels (so 4 conditions). Furthermore, I have one dependent variable (qualitative), also consisting of two levels. Now I want to make pairwise ...
0
votes
1answer
69 views

glm inflated error…why?

I'm pretty new to stats, so this may be dumb. I've been running a bunch of models on randomly generated data to try and develop my understanding of type 1 error. I've noticed that using ...
1
vote
1answer
96 views

Confidence Interval Coverage-error and Type I error

Could somebody explain to me the relationship between coverage error and type one errors in multiple comparisons testing, if there is one in fact? Does a coverage error occur when the true value of ...
1
vote
0answers
264 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...
2
votes
2answers
151 views

Comparing different conditions on a binomial distribution

I have some data where I have tested a binomial random variable under 4 conditions. The null hypothesis is that they all have equal means, alternative hypothesis is that one or more means differ from ...
1
vote
0answers
258 views

2x2x2 repeated measures anova: do I correct for type I error?

I have a 2x2x2 repeated measures design and am trying to decompose a 3-way interaction. As all my factors have 2 levels, I always reach the point that I am doing pairwise comparisons for the simple ...
6
votes
1answer
106 views

How to measure uncertainty of a parameter when false positives exist?

The main goal of my research is to measure the percentage of brown dwarf stars in the Pleiades star forming cluster that are actually double stars (i.e. the brown dwarf star has a companion brown ...
1
vote
0answers
210 views

Analyzing type I error rate in a simulation study using ANOVA-like methods

I've run a simulation study in order to determine type I error rate of a statistic.My simulation design includes threes factors as sample size (4 levels), test length or number of items (3 levels) and ...
1
vote
2answers
166 views

Three important issues

I don't understand three important issues : (1) Why do we calculate the test statistic under the null hypothesis ? (2) Why do we reject the null hypothesis when : p-value < significance level ? ...
0
votes
1answer
82 views

Can I make any tests statistically significant?

I am using a simple $\chi^2$ test to see if there is any impact of an independent variable (two charts) on the resulting choices (two choices). I found that as the number of sample increase, the ...
3
votes
2answers
495 views

Multiple resampling test/train dataset when choosing new models?

I have been reading several posts on testing multiple models on the same dataset, which can lead to problems controling type-1 errors. Mostly these posts have to do with data-mining on big datasets: ...
3
votes
1answer
758 views

Determining sample size with a proportion and binomial distribution

I am trying to learn some statistics using the book, Biometry by Sokal and Rohlf (3e). This is an exercise in the 5th chapter which covers probability, the binomial distribution, and Poisson ...
5
votes
1answer
340 views
1
vote
0answers
316 views

Calculating the size of Type 1 error, Type 2 error and power of the test

Let $X$ have a binomial distribution with parameter $n=5$ and $P\in [p:p=\frac{1}{4},\frac{1}{2}]$. The null hypothesis $H_{0}:P=\frac{1}{4}$ is rejected, and The alternative hypothesis ...
11
votes
1answer
4k views

How and when to use the Bonferroni adjustment

I have two questions regarding when to use a Bonferroni adjustment: Is it appropriate to use a Bonferroni adjustment in all cases of multiple testing? If one performs a test on a data set, then one ...
1
vote
1answer
369 views

Type 1 error Bayesian Stats

Is there anybody who can tell me or explain to me about finding the Type 1 error for a Bayesian clinical trial. Of a 100 patients I need 59 successful patients to exceed my posterior threshold. I ...