Rejecting the null hypothesis when it is true.

learn more… | top users | synonyms

1
vote
1answer
36 views

What is the empirical size of a test?

Now I am doing research of a proposed test statistic. I want to calculate the empirical sizes for different sample size of the proposed test statistic under the nominal type I error,such as 0.05. ...
0
votes
2answers
56 views

Calculate type I and II error - solution verification

There are 7 balls in urn. $Q$ of them are white and the rest are black. We have hypothesis $H_0:Q=3$ and $H_1:Q=5$. To test this we draw 2 balls (balls don't come back to the urn - i.e. they are drawn ...
0
votes
1answer
22 views

How to calculate theoretical quantiles of an odd dataset?

In order to calculate a set of theoretical quantiles I usually apply the following method: $\frac{1}{2. N} + \frac{x}{N}$ so here this makes: $\frac{1}{24} + \frac{6}{12}$ $\frac{1}{24} + ...
3
votes
2answers
49 views

How to calculate the probability of making a type 2 error?

Knowing the probability of having a type 1 error equals to $\alpha$ (significance). I think it might be incorrect to tell that if $\alpha = 20 $ the chances of making a type 2 error are equal to 80%. ...
1
vote
0answers
18 views

Inflated Type 1 error in glmer (for main effect but not interaction?)

I am doing simulations of type 1 error, power, and power' (power corrected for anti conservativity) for research on a specific application of (g)lmer, namely to small-N designs of longitudinal ...
1
vote
0answers
14 views

A situation where ignoring clustering optimises the Type I and Type II error rates?

I am interested in modelling clustered data with a small number of clusters as follows: $$Y_{ij} = β_0 + β_1X + u_i + e_{ij}$$ (where $_i$ = 1 to 3; $_j$ = 1 to 12); $Y_{ij}$ is our normally ...
0
votes
0answers
17 views

Second type error for difference in proportions test

I want to calculate the second type error Beta for a difference in proportions test, in a two populations scenario. Google search guides me to different tutorials to perform the test of proportions, ...
3
votes
2answers
39 views

Checking stability of a model in relation to experiment-wise error rate (philosophical)

I recently read an article that included a checklist aimed at improving the reporting of results in psychology. Among other things, they made the following suggestions: If observations are ...
3
votes
0answers
38 views

Confidence Bands vs. Simultaneous Confidence Intervals

This may be a dumb question, but when talking about multiple regression analyses are "simultaneous" confidence intervals and confidence "bands" the same thing? I'm still having trouble figuring this ...
1
vote
0answers
29 views

Estimating the false positive rate for a strictly standardized mean difference

Does anyone have a good source that clearly explains how to estimate the false positive rate or restricted false positive rate of the strictly standardized mean difference (SSMD)? I am trying to ...
4
votes
3answers
162 views

Does false discovery rate depend on the p-value or only on the alpha level?

Let's say I get a p-value of 0.001. I know that alpha level dictates the probability of a type I error, so if I get a result this significant, is my false discovery rate (FDR) lower than if I were to ...
1
vote
1answer
47 views

Dependence, independence, multiple testing, and alpha error correction

although I have studied psychology and feel I should know a lot more about issues such as this one, I have a quite basic question (or maybe not so basic question, considering all the fuss about this) ...
1
vote
0answers
84 views

Correcting for family-wise error rate with series of repeated measures ANOVA?

I am trying to make requested revisions to an accepted manuscript, and I am baffled by the following comment from a reviewer: "Eight hypothesis tests are reported in the final paragraph of the ...
0
votes
0answers
22 views

Using Likelihood Ratio Test to deal with heteroscedastic data results in unreliable results

Suppose $Y$ and $x$ are not related. Therefore the linear regression analysis should not reject the null hypothesis ($H_0: b=0$) in $E(Y) = a+bx$. Suppose the variance in $Y$ increase with $x$ (i.e., ...
3
votes
1answer
83 views

What is the name for this unintuitive result with OLS on a very “asymmetric” regressand, and how should it be addressed?

Say our sample consists of about a hundred Belgian (x = 0) and Swiss (x = 1) chocolate bars. We test them to see if they have safe (y = 1) or lethal (y = 0) levels of arsenic. As it turn out, 90% are ...
0
votes
1answer
49 views

Fairness of a Coin

A coin is tossed $40$ times. Define $T$ as the number of tails. i) Define the region of rejection by $|T-20|\geq5.$ Calculate $\alpha,$ the significance level --- $\displaystyle\alpha = P(y\leq15 ...
2
votes
1answer
64 views

How to estimate probability of $\geq$ n type I error in multiple testing comparison?

My question is about the calculation of the probability of making $\geq n$ type I error when $p$ independent statistical tests are made. I can calculate the probability of $\geq 1$ type I error with ...
3
votes
2answers
741 views

Is it a contradiction to do a hypothesis test on 1000 simulated datasets and never get a p value <0.05. Type 1 error question.

I am trying to calculate the type 1 error of a bootstrap hypothesis test procedure (won't go into the hypothesis test here). I ran the test on 1000 simulated datasets (simulated under the null ...
0
votes
1answer
158 views

Confidence interval and type 1 error cumulation

I have a sample of 15 subjects that did a test multiple times. Now I would like to see if the subjects differ in their mean performance. The usual approach would be a test for differences in means ...
4
votes
2answers
139 views

Why is the complement to Power not $\alpha$?

i) Wrongly rejecting $H_0$ is called a type I error (controlled by $\alpha$). ii) Wrongly accepting $H_0$ is called a type II error (the probability of which is indicated by $\beta$). iii) Power is ...
3
votes
0answers
41 views

Is the increase in Type I errors due to multi-sample t-tests overstated?

To illustrate, let's say you want to compare the means of 3 samples with t-tests. You would then have to run 3 tests to compare the means of each possible combination of samples. The explanations I ...
5
votes
1answer
263 views

How bad can heteroscedasticity be before causing problems?

I have two questions about heteroscedasticity in multiple regressions. According to my trusty textbook (Using Multivariate Statistics 2007, p.127), it says that deviations from ...
9
votes
3answers
330 views

Are probabilities of Type I and II errors negatively correlated?

In an elementary statistics class that I was a TA for, the professor stated that as the probability of a type I error $\alpha$ increases, the probability of a type II error $\beta$ decreases, and the ...
2
votes
0answers
69 views

Using simulation to estimate type-I error in Bayesian Tests

When doing a bayesian test, it is possible to estimate a "type-I error" of the test procedure by generating data from the null-hypothesis and running the bayesian test procedure several times. While ...
4
votes
2answers
169 views

Effects of blocking on type I and type II error rates

I am studying blocking in ANOVA and I am wondering about the following scenario. Suppose we did a Generalised randomised block design. Suppose SSBL = 0 and it also had not interaction effect with the ...
7
votes
3answers
715 views

Why is type I error not affected by different sample size - hypothesis testing? [duplicate]

I don't understand why the probability of getting a type I error when performing a hypothesis test, isn't affected. Increasing $n$ $\Rightarrow$ decreases standard deviation $\Rightarrow$ make the ...
8
votes
5answers
255 views

What is the relationship between $p$ values and Type I errors [duplicate]

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was ...
2
votes
0answers
125 views

rejection region for correlated bivariate normal

STATEMENT OF PROBLEM: Suppose $ \left( \begin{array}{ccc} \ Z_1 \\ Z_2 \end{array} \right)$ follows a Bivariate standard normal with covariance $ \rho $ $ \left( \begin{array}{ccc} \ Z_1 \\ Z_2 ...
3
votes
0answers
69 views

Does N-1 correction for Pearson's Chi-Square apply to r x c tables generally?

Campbell (2007) recommended what he called the "N-1" correction for chi-square tests performed on two-by-two contingency tables. He described the correction as: "the K. Pearson chi-squared test ...
8
votes
1answer
344 views

Low sample size: LR vs F - test

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. Currently I am comparing negative binomial ...
1
vote
3answers
221 views

What is the basis of setting critical p-value value in stepwise regression?

In statistical software like MINITAB and SAS, the default alpha value (critical p-value) is set as 0.15. I would like to know (1) if there is any statistic basis to set it as 0.15 and (2) if this is a ...
0
votes
1answer
22 views

error summary metric

I am working on trying to get a summary metric that summarizes the results of four models (M1:M5) and preferably ranges from zero to one, with one being the best model and zero being the worst model. ...
0
votes
2answers
81 views

Simulating violations of regression assumptions

I'm wondering if anyone could provide some code (preferably in R) which demonstrates violated assumptions leading to type 1 errors. Some concrete examples of errors arising from assumption violations ...
9
votes
1answer
218 views

Increased Type I error - GLM

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. In my field of research (ecotoxicology) ...
37
votes
5answers
1k views

Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
5
votes
2answers
133 views

Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple ...
1
vote
1answer
33 views

Critical region for difference in proportions

A new virus breaks out on a cruise ship. I want to test the hypothesis that males and females are equally likely to contract the virus. I am going to test 100 men and 100 women. Presumably if I find ...
1
vote
2answers
349 views

Examples for Type I and Type II errors

I was checking on Type I (reject a true H$_{0}$) and Type II (fail to reject a false H$_{0}$) errors during hypothesis testing and got to to know the definitions. But I was looking for where and how ...
3
votes
2answers
710 views

Is p-value also the false discovery rate?

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) It states "For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery ...
0
votes
1answer
576 views

post-hoc test after logistic regression with interaction. Risk higher for type 1 or type 2 error?

I have a large data set (>1000 obs) and i'm performing regressions tests, both linear and logistic, on a series of clinical outcomes. In this test I verify the effect of interactions between two cat ...
6
votes
2answers
355 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
1
vote
1answer
81 views

Is it possible to randomly sample from single data set (Monte Carlo style) to create new data sets?

Background I understand Monte Carlo methods only superficially, but I understand you can repeatedly randomly sample, with or without replacement, from your data set to estimate population parameters ...
1
vote
1answer
594 views

How to control for type-1-error inflation with multiple chi-squared tests?

I have two between-subject factors, each with two levels (so 4 conditions). Furthermore, I have one dependent variable (qualitative), also consisting of two levels. Now I want to make pairwise ...
0
votes
1answer
71 views

glm inflated error…why?

I'm pretty new to stats, so this may be dumb. I've been running a bunch of models on randomly generated data to try and develop my understanding of type 1 error. I've noticed that using ...
1
vote
1answer
117 views

Confidence Interval Coverage-error and Type I error

Could somebody explain to me the relationship between coverage error and type one errors in multiple comparisons testing, if there is one in fact? Does a coverage error occur when the true value of ...
1
vote
0answers
383 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...
2
votes
2answers
200 views

Comparing different conditions on a binomial distribution

I have some data where I have tested a binomial random variable under 4 conditions. The null hypothesis is that they all have equal means, alternative hypothesis is that one or more means differ from ...
1
vote
0answers
308 views

2x2x2 repeated measures anova: do I correct for type I error?

I have a 2x2x2 repeated measures design and am trying to decompose a 3-way interaction. As all my factors have 2 levels, I always reach the point that I am doing pairwise comparisons for the simple ...
6
votes
1answer
109 views

How to measure uncertainty of a parameter when false positives exist?

The main goal of my research is to measure the percentage of brown dwarf stars in the Pleiades star forming cluster that are actually double stars (i.e. the brown dwarf star has a companion brown ...
1
vote
0answers
267 views

Analyzing type I error rate in a simulation study using ANOVA-like methods

I've run a simulation study in order to determine type I error rate of a statistic.My simulation design includes threes factors as sample size (4 levels), test length or number of items (3 levels) and ...