Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [type-i-and-ii-errors]

Type I: rejecting the null hypothesis when it is true. Type II: not rejecting the null hypothesis when the alternative is true.

1
vote
0answers
13 views

Sample size formulae

In order to calculate sample size for hypothesis testing, we consider level of significance and power. For level of significance, two sided values ($Z_{\alpha/2}$) are taken into account, but for ...
0
votes
0answers
20 views

Can Bayes factors be an alternative to multiple comparisons corrections?

I am currently using p-values and Bayes factors to interpret my results. I have multiple comparisons so was going to perform a Bonferroni correction, however I am hesistant due to its highly ...
0
votes
0answers
25 views

Is it ever valid to choose test parameters based on an observed sample? [closed]

To illustrate, let me give a simple example. Suppose I will take some samples from a normal distribution with unknown mean $\mu$ and known variance $\sigma^2$, and for some particular relative error $...
0
votes
0answers
7 views

Sensitivity and Specificity of gaussian and negative binomial glm family

I am dealing with count data that is over-dispersed and hence I consider using the negative binomial family (with glm()) instead of the gaussian family, as the link function. To find out the ...
0
votes
1answer
34 views

Type I error question

I have this question from a previous exam, since it didn't worth much points I suspect it should be something tricky. How to achieve a statistical test where the Type I error risk is $0$? Is it ...
0
votes
0answers
22 views

type 1 error for mean

I was trying to do an exercise that says to calculate the Type 1 error for the test function $\delta$ given by $$\delta(X_1,...,X_n) = 1 ~~~~~~~if~~~~~ |\overline{X}|\geq Q,~~~~~~~~\delta(X_1,...,X_n) ...
0
votes
1answer
16 views

How to measure risk of a Type 2 error in A/B tests

I'm conducting A/B tests where outcomes are binary (success or failure in control and experiment populations). The assessment of the risk of a Type 1 error is relatively straightforward: given an ...
1
vote
0answers
22 views

Finding the probability of a type II error for a poisson distribution

A poisson distribution is being tested with the hypothesis H0: λ = 1.7 and H1: λ ≠ 1.7 The true value of the mean is 2.3. For the significant level 10% calculate: a )The critical region b) The ...
0
votes
1answer
16 views

Finding the probability of a type II error for a binomial distribution

For a binomial distribution, the hypothesis $H_0: p = 0.2$ and $H_1: p\ne0.2$ are tested at the $10%$ level. $20$ trials are performed and the critical region is $X = 0$ or $X > 7$. Calculate the ...
1
vote
0answers
46 views

Replicating Dickey Fuller? How did they know something was wrong? [cross posted on QF] [closed]

I am interested in testing if there is size distortion through simulations. I have recently been interested in replicating Dickey and Fuller (1979) and this source from another post helped a lot, here ...
2
votes
1answer
30 views

Sample size calculation for Kendall's Tau in reproducibility study

Can anyone help with justifying (or rubbishing) a couple of aspects of a sample size calculation for a grant proposal. A pilot study gave significant results for the correlation of a continuous (non-...
0
votes
0answers
11 views

Type II error for share price event study - what is the difference between one and several securities?

I have a question regarding the type II error for a specific event study. It is a case study (i.e. one observation) with daily share prices. I want to test whether event day abnormal returns are ...
0
votes
1answer
28 views

Type-I and Type-II errors example explanation

In the following example, can someone help understand how the value for $k$ (highlighted in yellow) is derived below? How is $z_\alpha$ introduced in there?
0
votes
0answers
29 views

Find type II error for event study share returns

I need help with calculating the type II error for a specific event study. It is a case study (i.e. one observation) and I want to test whether event day abnormal returns are different from zero. I ...
0
votes
1answer
54 views

Type I and Type II errors in Hypothesis Testing

I am confused about the last highlighted sentence regarding finding a subset S, for which BOTH Type I and Type II error probabilities are 0. For $P_\theta S$, which is the probability with which we ...
0
votes
0answers
32 views

Empirical Type I error from simulation different from $\alpha$ and changes based on sample size $n$

I am simulating LRT based on Chi-squared random variables. I know that the asymptotic distribution of LRT statistic follows chi-squared distribution with $df=$difference of the degrees of freedom ...
0
votes
1answer
18 views

Probability of Type II error for simple hypothesis

I'm working through Intro to Probability and Statistics... and I'm a little confused about how to calculate the probability of a type II error, particularly as it relates to a simple hypothesis test, ...
3
votes
3answers
451 views

Is there a constraint on the sum of the type-I & type II error probabilities?

Is it true that if $H_0$ and $H_a$ are complementary hypotheses of the Binomial trial, i.e., the negation of $H_0$ is $H_a$ then the type-I error $\alpha$ plus type-II error $\beta$ equals 1? Or is ...
2
votes
3answers
151 views

Does multicollinearity cause type I errors?

Citing Wikipedia on multicollinearity, One of the features of multicollinearity is that the standard errors of the affected coefficients tend to be large. In that case, the test of the hypothesis ...
3
votes
1answer
38 views

Can I perform a Chi Square test comparing the one year mortality of my 160 patients versus the 108000 patients of a cohort in the litterature?

I have exactly zero experience with statistics. I was interested in comparing 1 year mortality of patients $\ge$ 60 years old versus $<$ 60 in a cohort after a specific intervention. I performed ...
2
votes
1answer
47 views

Type I and Type II errors when varying the number of treatments

I have a hypothetical sample of n observations, and I want to test if there are significant differences across multiple treatments. I need to know how the number of treatments interfere in the type I ...
0
votes
0answers
17 views

Familywise error rate for multiple comparisons across multiple independent variables?

Is the familywise error rate of 1-(1-alpha)^k, where k is the number of tests performed still valid if not all comparisons are made using the same independent variable? Say, I have 3 categorical ...
9
votes
2answers
1k views

“Liberal” p-values?

My question is rather semantic. When a method routinely produces high p-values it is called conservative. Would you call the opposite, i.e. a method with a high type-II-error rate liberal?
0
votes
2answers
25 views

Minimize Type II errors in black-box testing

Okay so I have a function in the shape of a black box that I'd like to test. It's characteristics are: It gives either 0, 1 or -1 as output. I can query it easily, producing tens of thousands of ...
21
votes
1answer
936 views

Why the names Type 1, 2 error?

What is the motivation of introducing an additional level of indirection from the descriptive 'false positive' to the integer '1'? Is 'false positive' really too long?
0
votes
1answer
40 views

Calculate type 2 error using one data file

I am a bit confused on how to calculate type 2 error to check whether the sample data I am using is sufficient. I have a data file that I have used to build a machine learning model. This data file ...
0
votes
0answers
42 views

How to get Type II error probability from G*Power

I realize that having standard error of the mean, critical value, sample mean values from a simple a priori power test in the G*Power free download, one could use NORM.DIST in Excel (Argh!) following ...
0
votes
0answers
52 views

What is the smallest value of r that gives a test of size not exceeding 0.05?

I've been given a question that I don't understand. Question: Consider a test which rejects $H_0$ if the number of observed successes is $≥ r$ , where $r$ is some number. What is the smallest value ...
2
votes
1answer
37 views

What if we use this kind of region of rejection?

I'm wondering how do we decide the "shape" of region of rejection in a hypothesis testing problem. Let's take a two-tailed z-test as an example. What if we choose the weird-looking region of rejection ...
1
vote
1answer
49 views

How to calculate % information in longitudinal group sequential design

I'm looking at using group sequential design for a study that will treat subjects with drug or placebo and then follow them for a year. The subjects get measured every 3 months for the year under ...
0
votes
0answers
21 views

How to Work out Type 1 Error in Probability Distribution?

For (a), my answer is 'accept H0 if the sample mean = 0, otherwise reject'. Can anyone help me understand how to answer question (b)? I know that the P(Type 1 Error) = the significance level, but I'm ...
0
votes
1answer
54 views

why correct for multiple comparisons in a correlation table?

I often read that in a correlation matrix the p-values for all correlations in a NxN matrix should be adjusted for multiple comparisons (for example: Multiple comparisons for correlation matrix?). ...
0
votes
0answers
12 views

Error correction within factorial GLMMs

Below is toy-data and analysis for a bog-standard factorial longitudinal GLMM ...
0
votes
0answers
82 views

Bias-variance decomposition for a statistical test

A statistical estimator can be characterized – among others – by its bias and variance. These are different aspects, but if one is willing to accept a certain loss function (risk) than they can merged;...
0
votes
0answers
82 views

Type I error rate simulations and p-value

In the simulation below with two random Poisson numbers with lambda = 42, the simulated type I error rate is around 4%, but the experimenter has chosen alpha = 0.05 as the true expected value. Does it ...
2
votes
4answers
254 views

Known alpha and type I error rate simulation

I am not sure why running the following code in R does not result in a type I error rate of exactly 5% given alpha = 0.05 when set.seed is set to 1. Is there a reason why running the simulation below ...
17
votes
2answers
3k views

FPR (false positive rate) vs FDR (false discovery rate)

The following quote comes from the famous research paper Statistical significance for genome wide studies by Storey & Tibshirani (2003): For example, a false positive rate of 5% means that on ...
0
votes
0answers
48 views

In using backward elimination procedure how to control for type I error?

Use backward elimination procedure to decide which predictor variables can be dropped from the regression model. Control the type I error at = . 10 at each stage In using backward ...
11
votes
2answers
442 views

How to rigorously justify chosen false-positive/false-negative error rates and underlying cost ratio?

Context A group of social scientists and statisticians (Benjamin et al., 2017) have recently suggested that the typical false-positive rate ($\alpha$ = .05) used as a threshold for determining "...
2
votes
0answers
61 views

When can I use a permutation test to see if means of two distributions differ without Type I errors?

I am analyzing psych experiments which generally take the form of a list of outcome measures for the test and control groups, with roughly equal sample sizes, and I would like to compare their means ...
1
vote
1answer
91 views

Power function of a test,example

Consider a population with the pdf $N(\theta,1)$ where $\theta$ is unknown and hypothesis $H_0:\theta=5.5$ and $H_1:\theta=8$.Suppose that $\bar{X}=\frac{1}{9}\sum_{i=1}^9 X_i.$Reject $H_0$ iff $\bar{...
3
votes
1answer
366 views

Formulas to calculate power of a test. What am I doing wrong?

In Jay L. Devore 'Probability and Statistics for Engineering and the Sciences', he presents the following formula to retrieve β, when we consider a two-tailed test and the population mean is known. $\...
2
votes
2answers
408 views

Does Central Limit Theorem Apply to Bayesian inference?

In reading a Paper on Bayesian estimation, I came across a sentence that had me think: "Bayesian statistics is not based on large samples (i.e., the central limit theorem) and hence may produce ...
3
votes
3answers
300 views

Chi-squared test of independence: is a type II error by itself a bad thing?

Let us assume I sell a product that I have modified recently and I want to check if the old or the new version sells better. Therefore, I conduct an experiment in which a hundred people are offered ...
0
votes
0answers
27 views

Working on the topic of significance testing

I have a question as follows: Assuming the null hypothesis is true for all cases. If we perform multiple pairwise comparisons at the $\alpha = 0.05$ level, what is the probability of making type I ...
0
votes
1answer
20 views

Lower chance of type I error for A>B>C than for A>B?

Say I want to test the effect of two levels of my independent variable (IV) on some dependent variable (DV). For instance, administering a drug in a small vs large dosage, and then measuring reaction ...
59
votes
5answers
4k views

Why does collecting data until obtaining a significant result increase the Type I error rate?

I was wondering exactly why collecting data until a significant result (e.g., $p \lt .05$) is obtained (i.e., p-hacking) increases the Type I error rate? I would also highly appreciate an ...
0
votes
2answers
40 views

Type I/II error

In my analyses I had a clear hypothesis about a relationship Y ~ X1. In order to show that this is specific, I ran a control analyses with Y ~ X2. I showed that the relationship with X1 was ...
3
votes
0answers
385 views

“Inflated” type I error rate for correlation test when a variable has repeated values

For a given variable Y, if we independently generate a random variable X and test the correlation between Y and X, we know that the histogram of the p-values of the correlation test should show a ...
0
votes
0answers
37 views

Finding β for binomial experiment

$X\sim \text{Binomial}(5,P)$ Rejection region $X ≤1$ $H_\text{null}: P=0.6$ $H_\text{alt}: P<0.6$ Find $β$ at $P= 0.3$ I understand that ($β$ = not in the rejection region/H alt....