Rejecting the null hypothesis when it is true.

learn more… | top users | synonyms

1
vote
0answers
9 views

A strange finding with the logrank test statistics

When I try to confirm the type I error rate of the logrank test by simulation, I found something strange... Firstly, I generated 100 observations from exponential distribution with lamda1=1 as group 1 ...
1
vote
1answer
25 views

Heteroskedasticity and Type I, Type II Error

If one is conducting a hypothesis test for heteroskedasticity, would one generally consider a Type I or a Type II error more serious? What is the reasoning behind this?
0
votes
0answers
31 views

How to calculate FAR & FRR from Confusion Matrix?

For a biometric authentication system, I'm trying to compute the False Acceptance Rate (FAR) and False Rejection Rate (FRR) from the confusion matrix generated in Weka. I am using SVM with 10-folds ...
1
vote
1answer
16 views

Is alpha error inflation a problem with multiple ordered logistic regressions?

Following problem: I'm doing a survey where my independent variables are: (1) the perceived attractiveness of a product on a 7-Likert-scale and (2) the willingness to invest in such a product as a ...
2
votes
1answer
53 views

Why would you want a smaller critical region when testing large samples?

Is it because when you have a smaller critical region you have a smaller probability of the event happening?
1
vote
2answers
37 views

Significance level in sample size calculation and in final analysis

Imagine that for the purpose of a study a sample size is computed with the following formula for a given power $1-\beta$, difference in means $\epsilon$, standard deviation $\sigma$ and significance ...
0
votes
0answers
10 views

type 1 and type 2 errors for non-normal residuals in a regression

I did a regression and performed a normality tests on my residuals, which reject the Null Hypothesis that the residuals are normally distributed (the true distribution seems to be flatter with fatter ...
0
votes
1answer
37 views

Is there a concern for multiple comparisons when ranking instead of using significance?

Let's say I have many ANOVA tests on the same data using different factor variables each time. Then, I want to rank the F-statistics for all of those tests. Notice that I'm not interested in ...
1
vote
0answers
194 views

Type 1 and Type 2 Errors

A fabric manufacturer believes that the proportion of orders for raw material arriving late is p =0 .6. If a random sample of 10 orders shows that 3 or fewer arrived late, the hypothesis that p =0 .6 ...
4
votes
2answers
190 views

When is probability of type-I error less than the level of significance?

My book mentions $$\textrm{Probability of Type I Error}\le \textrm{Level of Significance}= \alpha$$ Now, I bore in mind that $\alpha$, the level of significance is described by $$\mathrm P(t\in \...
3
votes
3answers
61 views

Does the logic of “family-wise error” apply to effect size estimation?

Background: As I understand, family-wise error refers to the inflation of Type I error when performing multiple hypothesis tests. For example, if I were to perform multiple post-hoc comparisons ...
10
votes
2answers
241 views

Why aren't type II errors emphasized as much in statistical literature?

I have seen many cases where type I errors are accounted for (denoted by an alpha value) in various research articles. I have found it rare that a researcher will take into consideration the power, or ...
1
vote
1answer
228 views

What is the empirical size of a test?

Now I am doing research of a proposed test statistic. I want to calculate the empirical sizes for different sample size of the proposed test statistic under the nominal type I error,such as 0.05. ...
0
votes
2answers
82 views

Calculate type I and II error - solution verification

There are 7 balls in urn. $Q$ of them are white and the rest are black. We have hypothesis $H_0:Q=3$ and $H_1:Q=5$. To test this we draw 2 balls (balls don't come back to the urn - i.e. they are drawn ...
0
votes
1answer
25 views

How to calculate theoretical quantiles of an odd dataset?

In order to calculate a set of theoretical quantiles I usually apply the following method: $\frac{1}{2. N} + \frac{x}{N}$ so here this makes: $\frac{1}{24} + \frac{6}{12}$ $\frac{1}{24} + \frac{...
3
votes
2answers
79 views

How to calculate the probability of making a type 2 error?

Knowing the probability of having a type 1 error equals to $\alpha$ (significance). I think it might be incorrect to tell that if $\alpha = 20 $ the chances of making a type 2 error are equal to 80%. ...
1
vote
0answers
28 views

Inflated Type 1 error in glmer (for main effect but not interaction?)

I am doing simulations of type 1 error, power, and power' (power corrected for anti conservativity) for research on a specific application of (g)lmer, namely to small-N designs of longitudinal ...
1
vote
0answers
26 views

A situation where ignoring clustering optimises the Type I and Type II error rates?

I am interested in modelling clustered data with a small number of clusters as follows: $$Y_{ij} = β_0 + β_1X + u_i + e_{ij}$$ (where $_i$ = 1 to 3; $_j$ = 1 to 12); $Y_{ij}$ is our normally ...
0
votes
0answers
25 views

Second type error for difference in proportions test

I want to calculate the second type error Beta for a difference in proportions test, in a two populations scenario. Google search guides me to different tutorials to perform the test of proportions, ...
3
votes
2answers
44 views

Checking stability of a model in relation to experiment-wise error rate (philosophical)

I recently read an article that included a checklist aimed at improving the reporting of results in psychology. Among other things, they made the following suggestions: If observations are ...
3
votes
0answers
76 views

Confidence Bands vs. Simultaneous Confidence Intervals

This may be a dumb question, but when talking about multiple regression analyses are "simultaneous" confidence intervals and confidence "bands" the same thing? I'm still having trouble figuring this ...
1
vote
0answers
35 views

Estimating the false positive rate for a strictly standardized mean difference

Does anyone have a good source that clearly explains how to estimate the false positive rate or restricted false positive rate of the strictly standardized mean difference (SSMD)? I am trying to ...
4
votes
3answers
240 views

Does false discovery rate depend on the p-value or only on the alpha level?

Let's say I get a p-value of 0.001. I know that alpha level dictates the probability of a type I error, so if I get a result this significant, is my false discovery rate (FDR) lower than if I were to ...
1
vote
1answer
67 views

Dependence, independence, multiple testing, and alpha error correction

although I have studied psychology and feel I should know a lot more about issues such as this one, I have a quite basic question (or maybe not so basic question, considering all the fuss about this) ...
1
vote
0answers
117 views

Correcting for family-wise error rate with series of repeated measures ANOVA?

I am trying to make requested revisions to an accepted manuscript, and I am baffled by the following comment from a reviewer: "Eight hypothesis tests are reported in the final paragraph of the ...
3
votes
1answer
94 views

What is the name for this unintuitive result with OLS on a very “asymmetric” regressand, and how should it be addressed?

Say our sample consists of about a hundred Belgian (x = 0) and Swiss (x = 1) chocolate bars. We test them to see if they have safe (y = 1) or lethal (y = 0) levels of arsenic. As it turn out, 90% are ...
0
votes
1answer
51 views

Fairness of a Coin

A coin is tossed $40$ times. Define $T$ as the number of tails. i) Define the region of rejection by $|T-20|\geq5.$ Calculate $\alpha,$ the significance level --- $\displaystyle\alpha = P(y\leq15 \...
2
votes
1answer
78 views

How to estimate probability of $\geq$ n type I error in multiple testing comparison?

My question is about the calculation of the probability of making $\geq n$ type I error when $p$ independent statistical tests are made. I can calculate the probability of $\geq 1$ type I error with ...
3
votes
2answers
749 views

Is it a contradiction to do a hypothesis test on 1000 simulated datasets and never get a p value <0.05. Type 1 error question.

I am trying to calculate the type 1 error of a bootstrap hypothesis test procedure (won't go into the hypothesis test here). I ran the test on 1000 simulated datasets (simulated under the null ...
0
votes
1answer
179 views

Confidence interval and type 1 error cumulation

I have a sample of 15 subjects that did a test multiple times. Now I would like to see if the subjects differ in their mean performance. The usual approach would be a test for differences in means (e....
4
votes
2answers
195 views

Why is the complement to Power not $\alpha$?

i) Wrongly rejecting $H_0$ is called a type I error (controlled by $\alpha$). ii) Wrongly accepting $H_0$ is called a type II error (the probability of which is indicated by $\beta$). iii) Power is ...
3
votes
0answers
43 views

Is the increase in Type I errors due to multi-sample t-tests overstated?

To illustrate, let's say you want to compare the means of 3 samples with t-tests. You would then have to run 3 tests to compare the means of each possible combination of samples. The explanations I ...
5
votes
1answer
433 views

How bad can heteroscedasticity be before causing problems?

I have two questions about heteroscedasticity in multiple regressions. According to my trusty textbook (Using Multivariate Statistics 2007, p.127), it says that deviations from ...
9
votes
3answers
369 views

Are probabilities of Type I and II errors negatively correlated?

In an elementary statistics class that I was a TA for, the professor stated that as the probability of a type I error $\alpha$ increases, the probability of a type II error $\beta$ decreases, and the ...
2
votes
0answers
81 views

Using simulation to estimate type-I error in Bayesian Tests

When doing a bayesian test, it is possible to estimate a "type-I error" of the test procedure by generating data from the null-hypothesis and running the bayesian test procedure several times. While ...
4
votes
2answers
236 views

Effects of blocking on type I and type II error rates

I am studying blocking in ANOVA and I am wondering about the following scenario. Suppose we did a Generalised randomised block design. Suppose SSBL = 0 and it also had not interaction effect with the ...
7
votes
3answers
2k views

Why is type I error not affected by different sample size - hypothesis testing? [duplicate]

I don't understand why the probability of getting a type I error when performing a hypothesis test, isn't affected. Increasing $n$ $\Rightarrow$ decreases standard deviation $\Rightarrow$ make the ...
8
votes
5answers
269 views

What is the relationship between $p$ values and Type I errors [duplicate]

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was ...
2
votes
0answers
143 views

rejection region for correlated bivariate normal

STATEMENT OF PROBLEM: Suppose $ \left( \begin{array}{ccc} \ Z_1 \\ Z_2 \end{array} \right)$ follows a Bivariate standard normal with covariance $ \rho $ $ \left( \begin{array}{ccc} \ Z_1 \\ Z_2 \...
3
votes
0answers
106 views

Does N-1 correction for Pearson's Chi-Square apply to r x c tables generally?

Campbell (2007) recommended what he called the "N-1" correction for chi-square tests performed on two-by-two contingency tables. He described the correction as: "the K. Pearson chi-squared test ...
8
votes
1answer
405 views

Low sample size: LR vs F - test

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. Currently I am comparing negative binomial ...
2
votes
4answers
299 views

What is the basis of setting critical p-value value in stepwise regression?

In statistical software like MINITAB and SAS, the default alpha value (critical p-value) is set as 0.15. I would like to know (1) if there is any statistic basis to set it as 0.15 and (2) if this is a ...
0
votes
1answer
23 views

error summary metric

I am working on trying to get a summary metric that summarizes the results of four models (M1:M5) and preferably ranges from zero to one, with one being the best model and zero being the worst model. ...
0
votes
2answers
108 views

Simulating violations of regression assumptions

I'm wondering if anyone could provide some code (preferably in R) which demonstrates violated assumptions leading to type 1 errors. Some concrete examples of errors arising from assumption violations ...
15
votes
3answers
338 views

Negative-binomial GLM vs. log-transforming for count data: increased Type I error rate

Some of you might have read this nice paper: O’Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods in Ecology and Evolution 1:118–122. klick. In my field of research (ecotoxicology) we'...
41
votes
5answers
2k views

Is the “hybrid” between Fisher and Neyman-Pearson approaches to statistical testing really an “incoherent mishmash”?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these ...
5
votes
2answers
137 views

Why am I getting a 10-15% type I error rate for a 2 x 2 ANOVA?

While testing type I error rate using R, I found that I am getting higher than 5% false positives with a 2-way ANOVA. However, I thought one of the reasons for using an ANOVA (as opposed to multiple ...
1
vote
1answer
33 views

Critical region for difference in proportions

A new virus breaks out on a cruise ship. I want to test the hypothesis that males and females are equally likely to contract the virus. I am going to test 100 men and 100 women. Presumably if I find ...
1
vote
2answers
459 views

Examples for Type I and Type II errors

I was checking on Type I (reject a true H$_{0}$) and Type II (fail to reject a false H$_{0}$) errors during hypothesis testing and got to to know the definitions. But I was looking for where and how ...
4
votes
2answers
1k views

Is p-value also the false discovery rate?

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) It states "For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery ...