Skip to main content

Questions tagged [type-i-and-ii-errors]

Type I: rejecting the null hypothesis when it is true. Type II: not rejecting the null hypothesis when the alternative is true.

Filter by
Sorted by
Tagged with
2 votes
2 answers
46 views

type I error in multiple comparisons summary statistics

I understand type I error can increase if we run 'multiple comparisons'. But does that refer to comparisons within a multi-category variable (or a group variable)? or multiple analyses using the same ...
aqen's user avatar
  • 191
0 votes
1 answer
65 views

Probability of committing a type I error

Let $X_1, ... ,X_n$ be an iid sample from the Bern(p) distribution, with $p \in (0, 1)$. I want to test the null hypothesis $H_0: p = \frac{1}{2}$ against the alternative $H_1: p \in (0, 1) \setminus $...
JohnD's user avatar
  • 117
3 votes
1 answer
113 views

If we know that there were no type 1 errors in previous tests, should we still make corrections for multiplicity?

Suppose there's a study where an interim analysis was performed to evaluate efficacy with the intention of stopping the study. However, since the predefined success criterion wasn't achieved, the ...
user7064's user avatar
  • 2,245
0 votes
0 answers
26 views

In a mixed effects model, can leaving out the intercept-slope correlation parameter inflate type I error?

I am considering leaving out the intercept-slope correlation parameter in a mixed effects model to avoid convergence issues (i.e., in nlme::lme, ...
Evan's user avatar
  • 225
1 vote
0 answers
34 views

Robust standard errors leading to false positives [closed]

I have an odd scenario in my data analysis and I'm not sure what is causing it. I have a large set of tuples $(Y_1, X_i) \dots, (Y_N, X_N)$ where $Y_i$ is a random vector from some arbitrary ...
David Wang's user avatar
2 votes
1 answer
28 views

Should I normalise my dependent variable by a measured variable that I want to be constant?

This is a question about the validity of applying the results of a statistical test, and unfortunately will require some explanation of the experiment in question. Essentially, I want to measure how ...
Rachel Jones's user avatar
1 vote
2 answers
71 views

Mann-Whitney U test, compared to Binomial test

I was doing some simulations on the Mann-Whitney U test. I collected a number of U statistics (U is the smallest number of wins out of all pairwise contests between 2 samples: compare all values of ...
jginestet's user avatar
  • 837
1 vote
1 answer
54 views

Should we favor family-wise error rate control or statistical power?

To counteract the multiple comparison problem, people generally use procedures that ensure strong level alpha FWER control (e.g., Bonferroni, Holm). However, these procedures necessarilly produce a ...
user1363251's user avatar
0 votes
1 answer
52 views

Distribution for which one-sample t-test yields inflated type 1 error rates

Type 1 error rates of the one-sample t-test (two-tailed) can be inflated for example for data from the exponential distribution. However, I am looking for distributions which yield inflated type 1 ...
StAgur's user avatar
  • 1
1 vote
0 answers
60 views

Can we derive type 2 error probability from confidence interval?

If we can compute confidence intervals for an estimate $\hat{\theta}$, then we can also compute the p value for an associated hypothesis $\theta=\theta_0$, which is the probability of a type 1 error (...
feetwet's user avatar
  • 1,148
2 votes
1 answer
88 views

Bayesian stats and multiple tests

Are Bayesian models subject to the same problems as frequentist ones, where we cannot run a bunch of different models due to Type I error? For example, let's say I have a large data frame on airplanes,...
aeiche01's user avatar
0 votes
1 answer
52 views

How to perform a 2 sample Kolmogorov-Smirnov test controlling for a type II error instead of type 1

I have four completely different sets of two samples and want to check for each set if both samples come from the same distribution, so four Kolmogorov-Smirnov tests total. If it's going to be ...
Frank's user avatar
  • 11
2 votes
0 answers
27 views

Is there any valid approach to adjust the statistical power taking the inflation of type 1 error into account?

I'm conducting some simulation analysis under different correlation structures where H0: absence of interaction component in the model. I computed type 1 error rate under the null hypothesis at some ...
Dovini Jayasinghe's user avatar
3 votes
1 answer
112 views

The use of p-values in tests of d-separation

In his very helpful online book on structural equation modeling, Jon Lefcheck writes the following concerning d-separation tests for SEMs: Once the model is fit, statistical independence is assessed ...
dbspon's user avatar
  • 371
4 votes
1 answer
196 views

On the logic meaning behind rejecting the null hypothesis and its relantionship with Type I error

I understand that the p-value is defined as the probability to obtain a "more extreme" value of $w$ if $H_0$ is true, i.e. $p=P(|W| > |w| \ |H_0 $is true$)$ and the "significance ...
volperossa's user avatar
1 vote
0 answers
38 views

Magnitude of Type I error inflation related to error noise after model selection

I am investigating Type I error inflation for the one-sided test $$H_0:\beta_2=0$$ after one step of forward stepwise regression for the following model: $$Y=X_1+\beta_2X_2+\epsilon$$ where $\epsilon \...
Julia Mathis's user avatar
0 votes
1 answer
52 views

How can I compute the probability of committing a type 2 error?

Let $X_1,...X_n$ be an iid sample distributed as $\mathcal{N}(\mu,1)$. We have the following tests $$H_0: \mu=0~~~\text{vs.}~~~H_1:\mu\neq 0$$and we use the statistic $T=\frac{1}{n}\sum_{k=1}^n X_k$ ...
user1294729's user avatar
0 votes
0 answers
26 views

What is the effect of adding variables to an analysis on type 1 and type 2 error?

I am planning a study where we have a low number of observations. We know we need to control for at least two variables, but other variables also exist that we can control for. It seems to me that ...
Ryan Folks's user avatar
13 votes
2 answers
320 views

The probability of making a Type S error, and the average amount of magnification (type M error) as a function of power

When you filter on only viewing results with p<alpha, some of those "statistically significant" results go in the wrong direction. This is called a Type S error ("S" for sign). ...
Harvey Motulsky's user avatar
1 vote
0 answers
401 views

How to correct for multiple testing in linear mixed models?

I want to correct for alpha error accumulation for my 10 crossed linear mixed models. The random parameters are specified as follows: random intercepts for subject and stimulus and a random slope for ...
CapsLock's user avatar
2 votes
1 answer
74 views

Bonferroni correction: Whats exactly is meant by "multiple tests"? [duplicate]

The Bonferroni correction seems to be quite controversial. But I read again and again that it should be used for multiple tests. But what exactly are multiple tests? If I have three different data ...
Concetta's user avatar
0 votes
1 answer
237 views

post-hoc analysis for interaction terms in logistic regression: emtrends(), type I error

I built a logistic regression, including the followed variables. And I tried to compare effects of temp among group and habitat using emtrends() in ...
pineapple159's user avatar
2 votes
1 answer
173 views

Visualizing how the 2 sample Z test works

In my stats class, we are talking about the 2 proportion $Z$-test, which compares two sample proportions. The test statistic is given as: $$ Z = \frac{\hat{p_1}-\hat{p_2}}{\sqrt{\hat{p_c}(1-\hat{p_c})...
Max0815's user avatar
  • 145
0 votes
0 answers
63 views

Type-2 errors in the likelihood-ratio/G-test?

I am quite new to hypothesis testing, and I am currently trying to familiarize myself with the G-test for independence. In my research, I came up with a question for which I could not fit an answer ...
J.Galt's user avatar
  • 555
2 votes
2 answers
70 views

In a muddle with Paired Samples t-tests on pre vs during vs post

I have used paired samples t-tests to compare 3x time-point measurements, A/ B/ C. A and B t(102)=4.988 p=.000 A and C t(102)=4.939 p=.000 B and C t(102)=.346 p=.730 A, B, and C were all significant ...
Stats_Apprentice's user avatar
1 vote
0 answers
42 views

Need help with Type 1 error and large number of chi squared results

I have a received comment for my research that addresses concern on the 'volume of null hypothesis test' being performed throughout my study and if any corrections are being done to account for 'an ...
elcielo0's user avatar
2 votes
2 answers
97 views

What does it mean for tests to be independent?

When reading about cumulation if type-1 Error, the sentence "for independent statistical tests" occures alot, now I was wondering what this is actually means. Since tests are also random ...
QED's user avatar
  • 163
1 vote
1 answer
86 views

Questions regarding power of test and type II error

I´m preparing for a lecture in decision theory and I´m a little bit confused by the notation used by my prof. On the first slide under remark 3.2 point v) its written, that $\beta(\varphi)$ is equal ...
this_is_not_easy's user avatar
1 vote
1 answer
131 views

What is the distribution and standard deviation of the number of Type I errors in hypothesis testing?

My question is from the book "An introduction to statistical learning" (2nd edition), chapter 13, exercise 2. Suppose that we test m hypotheses, and control the Type I error for each ...
Juan's user avatar
  • 57
0 votes
0 answers
26 views

What does look back mean observational studies?

Consider I am doing observational study either case control or cohort in either of the following cases. Case. 1 Suppose I found enrollment/rate of population with disease is too low where disease ...
user45765's user avatar
  • 1,446
1 vote
0 answers
44 views

In linear regression, does the t-test of significance satisfy $\sup_{\beta\in\Theta_0}Pr(\textrm{Reject}\ H_0: \beta_j=0)=\alpha$ asymptotically?

Suppose I have a linear regression $Y=\beta_0+\beta_1X_1+\beta_2X_2+U\equiv X'\beta+U$, where $E(U|X)=0$. After we have found the least squares estimator of $\beta\equiv[\beta_0,\beta_1,\beta_2]$ as $\...
ExcitedSnail's user avatar
  • 2,884
0 votes
0 answers
37 views

Methods to control for false omission rate?

What methods are available to control for false omission rate in multiple comparisons, similar to the way Benjamini-Hochberg controls for false discovery rate? The goal is to filter out obvious ...
goweon's user avatar
  • 253
1 vote
0 answers
123 views

Calculating Type 2 Error

The question given is: When performing a test regarding the population mean, $\mu$, we decide not to reject $H_0$ when $\bar{X} > 3.$ It is known that $n = 50.$ If the true value for $\mu$ is $\mu ...
Math.anony's user avatar
1 vote
1 answer
325 views

Equivalent to Dunnett's test for binomial endpoint?

I have a trial comparing 2 treatments vs a control group. Typically one would use Dunnett's test for this situation, but the outcome measure is binomial. It seems strange to me but I've looked for ...
Bosley's user avatar
  • 549
4 votes
1 answer
296 views

History of terms type 1 error and type 2 error?

The terms "type I" (or "alpha) and "type II" (or "beta) error, to denote false positive and false negative, are often used. What is the history of those terms?
deMoivre's user avatar
2 votes
1 answer
108 views

Does p-hacking affect the Type II (False Negative) error rate?

I know that p-hacking (also know as data dredging) increases the Type I error rate as exemplified by this XKCD example. My question is whether it influences the Type II error rate. I'm not sure ...
sloth's user avatar
  • 123
0 votes
0 answers
31 views

$H_2$, its might and Type III error

I've recently found myself wondering about the following thing in hypothesis testing: We can test some null hypothesis $H_0$ and reject it based on its p-value. This is akin to accepting (shreek!!!) ...
user11130854's user avatar
2 votes
0 answers
319 views

Are Type I and Type II errors in multiclass problem appear to be the same?

As far as I know from binary classification FP error is a type 1 error FN error is a type 2 error I have this confusion matrix generated: And here I found how to read this confusion matrix: As you ...
Elvin's user avatar
  • 121
2 votes
1 answer
315 views

Is it necessary to correct p-values for multiple comparisons in multivariate regression?

In this suspiciously brief comment on another article, the author suggests that analysts should only correct for multiple comparisons in univariate regression when the predictor has more than two ...
llewmills's user avatar
  • 2,161
2 votes
1 answer
106 views

Should one examine the cross-correlation plot to rule out performing Granger causality test?

Should one look at cross-correlation plot before performing Granger causality test to avoid type I errors? If we can't find any dependence between two series from the cross-correlation plot, then ...
user avatar
3 votes
1 answer
110 views

Two confusion matrices

In a machine learning context, I am working on a binary classification problem. There is a source of truth $T$ for labels, and a labeling process $A$ which is not perfect and makes errors compared to $...
Frank's user avatar
  • 1,686
2 votes
0 answers
597 views

For small sample sizes, is jackknife superior at controlling Type-I error compared to bootstrap?

This question is motivated by the post here: Can bootstrap be seen as a "cure" for the small sample size? In the referenced post, we see that the bootstrap approach does not control type-1 ...
3487564's user avatar
  • 21
0 votes
2 answers
357 views

Is confounding a source of Type I errors?

I would be interested to explore the potential sources of Type I errors in randomised controlled trials. A key question in this respect is whether confounding can be understood as one such source.
Purist's user avatar
  • 31
2 votes
1 answer
372 views

Is my way of deriving a statistical test from Hoeffding's inequality correct?

I'm trying to deduce from samples of observations from two finite sets of random variables $X_{1}, ..., X_{n}$ and $Y_{1}, ..., Y_{m}$ that the expected values of the average of those random variables ...
Daviiid's user avatar
  • 155
0 votes
0 answers
53 views

Testing $H_0: \theta \leq 0$ versus $H_1: \theta > 0$: Find $c$ in rejection region so that probability of a type I error is $\alpha$

My problem is as follows: Let $X_1,\ldots,X_n$ be a random sample from a $N(\theta,1)$ distribution, and consider testing $H_0: \theta \leq 0$ versus $H_1: \theta > 0$. Let $\Omega = \{\mathbf{X}: ...
Leonidas's user avatar
  • 121
2 votes
1 answer
206 views

How to simulate type I error for random-effects model?

Building upon this post How to simulate type I error and type II error. I would like to simulate type I error for a random-effects model I generated. The statistic of interest is standard deviations ...
cliu's user avatar
  • 211
0 votes
1 answer
347 views

simple vs simple hypothesis, finding sample size given type 1 and 2 error probabilities

The problem is: It is decided to investigate the hypotheses H0 : μ = €678 and H1 : μ = €720. Assuming that the prices are normally distributed with standard deviation €100, what sample size must be ...
seems-to-work-enjoyer's user avatar
2 votes
0 answers
75 views

Measurement error and type I. and type II. error [closed]

Is there a relationship between measurement error (1.) both systematic and random, (2.) systematic only and (3.) random only in terms of tendency of results of statistical tests to be biased towards ...
Nothingman's user avatar
2 votes
1 answer
43 views

Simple question on probability computation for multiple comparisons

In class, we illustrated the multiple comparisons problem through a simple example (no, not the xkcd jelly beans comic). Essentially, with $N$ independent tests each at level $\alpha$, we said the ...
Golden_Ratio's user avatar
0 votes
0 answers
236 views

Question about power of FDR and FWE

I understand that false discovery rate (FDR) is $\leq$ familywise error rate (FWE). I have read this means that controlling FDR is hence "more powerful" than controlling FWE. What does ...
Golden_Ratio's user avatar

1
2 3 4 5 6