Rejecting the null hypothesis when it is true.

learn more… | top users | synonyms

1
vote
0answers
94 views

Type 1 and Type 2 Errors

A fabric manufacturer believes that the proportion of orders for raw material arriving late is p =0 .6. If a random sample of 10 orders shows that 3 or fewer arrived late, the hypothesis that p =0 .6 ...
3
votes
2answers
181 views

When is probability of type-I error less than the level of significance?

My book mentions $$\textrm{Probability of Type I Error}\le \textrm{Level of Significance}= \alpha$$ Now, I bore in mind that $\alpha$, the level of significance is described by $$\mathrm P(t\in ...
3
votes
3answers
54 views

Does the logic of “family-wise error” apply to effect size estimation?

Background: As I understand, family-wise error refers to the inflation of Type I error when performing multiple hypothesis tests. For example, if I were to perform multiple post-hoc comparisons ...
10
votes
2answers
232 views

Why aren't type II errors emphasized as much in statistical literature?

I have seen many cases where type I errors are accounted for (denoted by an alpha value) in various research articles. I have found it rare that a researcher will take into consideration the power, or ...
1
vote
1answer
136 views

What is the empirical size of a test?

Now I am doing research of a proposed test statistic. I want to calculate the empirical sizes for different sample size of the proposed test statistic under the nominal type I error,such as 0.05. ...
0
votes
2answers
73 views

Calculate type I and II error - solution verification

There are 7 balls in urn. $Q$ of them are white and the rest are black. We have hypothesis $H_0:Q=3$ and $H_1:Q=5$. To test this we draw 2 balls (balls don't come back to the urn - i.e. they are drawn ...
0
votes
1answer
24 views

How to calculate theoretical quantiles of an odd dataset?

In order to calculate a set of theoretical quantiles I usually apply the following method: $\frac{1}{2. N} + \frac{x}{N}$ so here this makes: $\frac{1}{24} + \frac{6}{12}$ $\frac{1}{24} + ...
3
votes
2answers
66 views

How to calculate the probability of making a type 2 error?

Knowing the probability of having a type 1 error equals to $\alpha$ (significance). I think it might be incorrect to tell that if $\alpha = 20 $ the chances of making a type 2 error are equal to 80%. ...
1
vote
0answers
21 views

Inflated Type 1 error in glmer (for main effect but not interaction?)

I am doing simulations of type 1 error, power, and power' (power corrected for anti conservativity) for research on a specific application of (g)lmer, namely to small-N designs of longitudinal ...
1
vote
0answers
19 views

A situation where ignoring clustering optimises the Type I and Type II error rates?

I am interested in modelling clustered data with a small number of clusters as follows: $$Y_{ij} = β_0 + β_1X + u_i + e_{ij}$$ (where $_i$ = 1 to 3; $_j$ = 1 to 12); $Y_{ij}$ is our normally ...
0
votes
0answers
22 views

Second type error for difference in proportions test

I want to calculate the second type error Beta for a difference in proportions test, in a two populations scenario. Google search guides me to different tutorials to perform the test of proportions, ...
3
votes
2answers
42 views

Checking stability of a model in relation to experiment-wise error rate (philosophical)

I recently read an article that included a checklist aimed at improving the reporting of results in psychology. Among other things, they made the following suggestions: If observations are ...
3
votes
0answers
60 views

Confidence Bands vs. Simultaneous Confidence Intervals

This may be a dumb question, but when talking about multiple regression analyses are "simultaneous" confidence intervals and confidence "bands" the same thing? I'm still having trouble figuring this ...
1
vote
0answers
31 views

Estimating the false positive rate for a strictly standardized mean difference

Does anyone have a good source that clearly explains how to estimate the false positive rate or restricted false positive rate of the strictly standardized mean difference (SSMD)? I am trying to ...
4
votes
3answers
202 views

Does false discovery rate depend on the p-value or only on the alpha level?

Let's say I get a p-value of 0.001. I know that alpha level dictates the probability of a type I error, so if I get a result this significant, is my false discovery rate (FDR) lower than if I were to ...
1
vote
1answer
61 views

Dependence, independence, multiple testing, and alpha error correction

although I have studied psychology and feel I should know a lot more about issues such as this one, I have a quite basic question (or maybe not so basic question, considering all the fuss about this) ...
1
vote
0answers
95 views

Correcting for family-wise error rate with series of repeated measures ANOVA?

I am trying to make requested revisions to an accepted manuscript, and I am baffled by the following comment from a reviewer: "Eight hypothesis tests are reported in the final paragraph of the ...
0
votes
0answers
25 views

Using Likelihood Ratio Test to deal with heteroscedastic data results in unreliable results

Suppose $Y$ and $x$ are not related. Therefore the linear regression analysis should not reject the null hypothesis ($H_0: b=0$) in $E(Y) = a+bx$. Suppose the variance in $Y$ increase with $x$ (i.e., ...
3
votes
1answer
93 views

What is the name for this unintuitive result with OLS on a very “asymmetric” regressand, and how should it be addressed?

Say our sample consists of about a hundred Belgian (x = 0) and Swiss (x = 1) chocolate bars. We test them to see if they have safe (y = 1) or lethal (y = 0) levels of arsenic. As it turn out, 90% are ...
0
votes
1answer
49 views

Fairness of a Coin

A coin is tossed $40$ times. Define $T$ as the number of tails. i) Define the region of rejection by $|T-20|\geq5.$ Calculate $\alpha,$ the significance level --- $\displaystyle\alpha = P(y\leq15 ...
2
votes
1answer
76 views

How to estimate probability of $\geq$ n type I error in multiple testing comparison?

My question is about the calculation of the probability of making $\geq n$ type I error when $p$ independent statistical tests are made. I can calculate the probability of $\geq 1$ type I error with ...
3
votes
2answers
743 views

Is it a contradiction to do a hypothesis test on 1000 simulated datasets and never get a p value <0.05. Type 1 error question.

I am trying to calculate the type 1 error of a bootstrap hypothesis test procedure (won't go into the hypothesis test here). I ran the test on 1000 simulated datasets (simulated under the null ...
0
votes
1answer
176 views

Confidence interval and type 1 error cumulation

I have a sample of 15 subjects that did a test multiple times. Now I would like to see if the subjects differ in their mean performance. The usual approach would be a test for differences in means ...
4
votes
2answers
175 views

Why is the complement to Power not $\alpha$?

i) Wrongly rejecting $H_0$ is called a type I error (controlled by $\alpha$). ii) Wrongly accepting $H_0$ is called a type II error (the probability of which is indicated by $\beta$). iii) Power is ...
3
votes
0answers
43 views

Is the increase in Type I errors due to multi-sample t-tests overstated?

To illustrate, let's say you want to compare the means of 3 samples with t-tests. You would then have to run 3 tests to compare the means of each possible combination of samples. The explanations I ...
5
votes
1answer
360 views

How bad can heteroscedasticity be before causing problems?

I have two questions about heteroscedasticity in multiple regressions. According to my trusty textbook (Using Multivariate Statistics 2007, p.127), it says that deviations from ...
9
votes
3answers
355 views

Are probabilities of Type I and II errors negatively correlated?

In an elementary statistics class that I was a TA for, the professor stated that as the probability of a type I error $\alpha$ increases, the probability of a type II error $\beta$ decreases, and the ...
2
votes
0answers
75 views

Using simulation to estimate type-I error in Bayesian Tests

When doing a bayesian test, it is possible to estimate a "type-I error" of the test procedure by generating data from the null-hypothesis and running the bayesian test procedure several times. While ...
4
votes
2answers
204 views

Effects of blocking on type I and type II error rates

I am studying blocking in ANOVA and I am wondering about the following scenario. Suppose we did a Generalised randomised block design. Suppose SSBL = 0 and it also had not interaction effect with the ...
7
votes
3answers
1k views

Why is type I error not affected by different sample size - hypothesis testing? [duplicate]

I don't understand why the probability of getting a type I error when performing a hypothesis test, isn't affected. Increasing $n$ $\Rightarrow$ decreases standard deviation $\Rightarrow$ make the ...
8
votes
5answers
260 views

What is the relationship between $p$ values and Type I errors [duplicate]

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was ...
2
votes
0answers
137 views

rejection region for correlated bivariate normal

STATEMENT OF PROBLEM: Suppose $ \left( \begin{array}{ccc} \ Z_1 \\ Z_2 \end{array} \right)$ follows a Bivariate standard normal with covariance $ \rho $ $ \left( \begin{array}{ccc} \ Z_1 \\ Z_2 ...
3
votes
0answers
91 views

Does N-1 correction for Pearson's Chi-Square apply to r x c tables generally?

Campbell (2007) recommended what he called the "N-1" correction for chi-square tests performed on two-by-two contingency tables. He described the correction as: "the K. Pearson chi-squared test ...
8
votes
1answer
382 views

Low sample size: LR vs F - test

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. Currently I am comparing negative binomial ...
2
votes
4answers
263 views

What is the basis of setting critical p-value value in stepwise regression?

In statistical software like MINITAB and SAS, the default alpha value (critical p-value) is set as 0.15. I would like to know (1) if there is any statistic basis to set it as 0.15 and (2) if this is a ...
0
votes
1answer
23 views

error summary metric

I am working on trying to get a summary metric that summarizes the results of four models (M1:M5) and preferably ranges from zero to one, with one being the best model and zero being the worst model. ...
0
votes
2answers
101 views

Simulating violations of regression assumptions

I'm wondering if anyone could provide some code (preferably in R) which demonstrates violated assumptions leading to type 1 errors. Some concrete examples of errors arising from assumption violations ...
9
votes
1answer
228 views

Increased Type I error - GLM

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. In my field of research (ecotoxicology) ...
39
votes
5answers
2k views

Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
5
votes
2answers
134 views

Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple ...
1
vote
1answer
33 views

Critical region for difference in proportions

A new virus breaks out on a cruise ship. I want to test the hypothesis that males and females are equally likely to contract the virus. I am going to test 100 men and 100 women. Presumably if I find ...
1
vote
2answers
429 views

Examples for Type I and Type II errors

I was checking on Type I (reject a true H$_{0}$) and Type II (fail to reject a false H$_{0}$) errors during hypothesis testing and got to to know the definitions. But I was looking for where and how ...
3
votes
2answers
989 views

Is p-value also the false discovery rate?

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) It states "For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery ...
0
votes
1answer
685 views

post-hoc test after logistic regression with interaction. Risk higher for type 1 or type 2 error?

I have a large data set (>1000 obs) and i'm performing regressions tests, both linear and logistic, on a series of clinical outcomes. In this test I verify the effect of interactions between two cat ...
6
votes
2answers
416 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
1
vote
1answer
86 views

Is it possible to randomly sample from single data set (Monte Carlo style) to create new data sets?

Background I understand Monte Carlo methods only superficially, but I understand you can repeatedly randomly sample, with or without replacement, from your data set to estimate population parameters ...
1
vote
1answer
653 views

How to control for type-1-error inflation with multiple chi-squared tests?

I have two between-subject factors, each with two levels (so 4 conditions). Furthermore, I have one dependent variable (qualitative), also consisting of two levels. Now I want to make pairwise ...
0
votes
1answer
71 views

glm inflated error…why?

I'm pretty new to stats, so this may be dumb. I've been running a bunch of models on randomly generated data to try and develop my understanding of type 1 error. I've noticed that using ...
1
vote
1answer
143 views

Confidence Interval Coverage-error and Type I error

Could somebody explain to me the relationship between coverage error and type one errors in multiple comparisons testing, if there is one in fact? Does a coverage error occur when the true value of ...
1
vote
0answers
459 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...