Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Type I: rejecting the null hypothesis when it is true. Type II: not rejecting the null hypothesis when the alternative is true.

0
votes
0answers
14 views

How can we predict reliability for type 1 censored data?

I am working on a somewhat challenging problem. I have data for a large set of equipment. My main research question is: How many units will fail in each future month in the coming decade? So far I ...
0
votes
0answers
12 views

How to Control the Type-1 Error Rate for T-Tests Used to Understand Sig. Interaction Effects

Basically, I ran a repeated measures ANOVA between groups R and D with one within-subjects factor with 2 levels. A significant interaction was found. To understand what was driving the interaction I ...
0
votes
0answers
19 views

Commenting on Type 1 Error rates [closed]

After plotting the CDF of the probabilities of type 1 error for a test at alpha=0.05 and n=15 i am getting the given plot. It it ok to say "there is evidence that the Type I error rate of the test has ...
3
votes
1answer
36 views

Can I perform a Chi Square test comparing the one year mortality of my 160 patients versus the 108000 patients of a cohort in the litterature?

I have exactly zero experience with statistics. I was interested in comparing 1 year mortality of patients $\ge$ 60 years old versus $<$ 60 in a cohort after a specific intervention. I performed ...
2
votes
1answer
31 views

Type I and Type II errors when varying the number of treatments

I have a hypothetical sample of n observations, and I want to test if there are significant differences across multiple treatments. I need to know how the number of treatments interfere in the type I ...
0
votes
0answers
8 views

Familywise error rate for multiple comparisons across multiple independent variables?

Is the familywise error rate of 1-(1-alpha)^k, where k is the number of tests performed still valid if not all comparisons are made using the same independent variable? Say, I have 3 categorical ...
9
votes
2answers
882 views

“Liberal” p-values?

My question is rather semantic. When a method routinely produces high p-values it is called conservative. Would you call the opposite, i.e. a method with a high type-II-error rate liberal?
0
votes
2answers
23 views

Minimize Type II errors in black-box testing

Okay so I have a function in the shape of a black box that I'd like to test. It's characteristics are: It gives either 0, 1 or -1 as output. I can query it easily, producing tens of thousands of ...
20
votes
1answer
879 views

Why the names Type 1, 2 error?

What is the motivation of introducing an additional level of indirection from the descriptive 'false positive' to the integer '1'? Is 'false positive' really too long?
0
votes
1answer
32 views

Calculate type 2 error using one data file

I am a bit confused on how to calculate type 2 error to check whether the sample data I am using is sufficient. I have a data file that I have used to build a machine learning model. This data file ...
0
votes
0answers
23 views

How to get Type II error probability from G*Power

I realize that having standard error of the mean, critical value, sample mean values from a simple a priori power test in the G*Power free download, one could use NORM.DIST in Excel (Argh!) following ...
0
votes
0answers
39 views

What is the smallest value of r that gives a test of size not exceeding 0.05?

I've been given a question that I don't understand. Question: Consider a test which rejects $H_0$ if the number of observed successes is $≥ r$ , where $r$ is some number. What is the smallest value ...
2
votes
1answer
33 views

What if we use this kind of region of rejection?

I'm wondering how do we decide the "shape" of region of rejection in a hypothesis testing problem. Let's take a two-tailed z-test as an example. What if we choose the weird-looking region of rejection ...
1
vote
1answer
23 views

How to calculate % information in longitudinal group sequential design

I'm looking at using group sequential design for a study that will treat subjects with drug or placebo and then follow them for a year. The subjects get measured every 3 months for the year under ...
0
votes
0answers
21 views

How to Work out Type 1 Error in Probability Distribution?

For (a), my answer is 'accept H0 if the sample mean = 0, otherwise reject'. Can anyone help me understand how to answer question (b)? I know that the P(Type 1 Error) = the significance level, but I'm ...
0
votes
1answer
35 views

why correct for multiple comparisons in a correlation table?

I often read that in a correlation matrix the p-values for all correlations in a NxN matrix should be adjusted for multiple comparisons (for example: Multiple comparisons for correlation matrix?). ...
0
votes
0answers
12 views

Error correction within factorial GLMMs

Below is toy-data and analysis for a bog-standard factorial longitudinal GLMM ...
0
votes
0answers
73 views

Bias-variance decomposition for a statistical test

A statistical estimator can be characterized – among others – by its bias and variance. These are different aspects, but if one is willing to accept a certain loss function (risk) than they can merged;...
0
votes
0answers
66 views

Type I error rate simulations and p-value

In the simulation below with two random Poisson numbers with lambda = 42, the simulated type I error rate is around 4%, but the experimenter has chosen alpha = 0.05 as the true expected value. Does it ...
2
votes
4answers
134 views

Known alpha and type I error rate simulation

I am not sure why running the following code in R does not result in a type I error rate of exactly 5% given alpha = 0.05 when set.seed is set to 1. Is there a reason why running the simulation below ...
15
votes
2answers
1k views

FPR (false positive rate) vs FDR (false discovery rate)

The following quote comes from the famous research paper Statistical significance for genome wide studies by Storey & Tibshirani (2003): For example, a false positive rate of 5% means that on ...
0
votes
0answers
29 views

In using backward elimination procedure how to control for type I error?

Use backward elimination procedure to decide which predictor variables can be dropped from the regression model. Control the type I error at = . 10 at each stage In using backward ...
10
votes
2answers
432 views

How to rigorously justify chosen false-positive/false-negative error rates and underlying cost ratio?

Context A group of social scientists and statisticians (Benjamin et al., 2017) have recently suggested that the typical false-positive rate ($\alpha$ = .05) used as a threshold for determining "...
2
votes
0answers
45 views

When can I use a permutation test to see if means of two distributions differ without Type I errors?

I am analyzing psych experiments which generally take the form of a list of outcome measures for the test and control groups, with roughly equal sample sizes, and I would like to compare their means ...
1
vote
1answer
77 views

Power function of a test,example

Consider a population with the pdf $N(\theta,1)$ where $\theta$ is unknown and hypothesis $H_0:\theta=5.5$ and $H_1:\theta=8$.Suppose that $\bar{X}=\frac{1}{9}\sum_{i=1}^9 X_i.$Reject $H_0$ iff $\bar{...
0
votes
0answers
27 views

Is t-test the right test to use in this case?

I have two batches of 10 samples each of nominally the same component, each batch made via a different manufacturing technique ("old technique" and "new technique", where the new technique is cheaper ...
0
votes
0answers
24 views

How do I compare models with different scope?

I'm looking for any research papers in the Statistics literature or any information, in general, about comparing the performance of two different models when one of them imposes more severe ...
3
votes
1answer
322 views

Formulas to calculate power of a test. What am I doing wrong?

In Jay L. Devore 'Probability and Statistics for Engineering and the Sciences', he presents the following formula to retrieve β, when we consider a two-tailed test and the population mean is known. $\...
2
votes
2answers
304 views

Does Central Limit Theorem Apply to Bayesian inference?

In reading a Paper on Bayesian estimation, I came across a sentence that had me think: "Bayesian statistics is not based on large samples (i.e., the central limit theorem) and hence may produce ...
3
votes
3answers
230 views

Chi-squared test of independence: is a type II error by itself a bad thing?

Let us assume I sell a product that I have modified recently and I want to check if the old or the new version sells better. Therefore, I conduct an experiment in which a hundred people are offered ...
0
votes
0answers
27 views

Working on the topic of significance testing

I have a question as follows: Assuming the null hypothesis is true for all cases. If we perform multiple pairwise comparisons at the $\alpha = 0.05$ level, what is the probability of making type I ...
0
votes
1answer
19 views

Lower chance of type I error for A>B>C than for A>B?

Say I want to test the effect of two levels of my independent variable (IV) on some dependent variable (DV). For instance, administering a drug in a small vs large dosage, and then measuring reaction ...
0
votes
0answers
43 views

Addressing variability in AB test - assessing AA test results

My goal is to determine whether my AB testing platform has an acceptable false positive rate, and how I can take such data into consideration when analyzing tests in the future. I have run 15 AA test ...
59
votes
5answers
4k views

Why does collecting data until obtaining a significant result increase the Type I error rate?

I was wondering exactly why collecting data until a significant result (e.g., $p \lt .05$) is obtained (i.e., p-hacking) increases the Type I error rate? I would also highly appreciate an ...
0
votes
2answers
39 views

Type I/II error

In my analyses I had a clear hypothesis about a relationship Y ~ X1. In order to show that this is specific, I ran a control analyses with Y ~ X2. I showed that the relationship with X1 was ...
3
votes
0answers
291 views

“Inflated” type I error rate for correlation test when a variable has repeated values

For a given variable Y, if we independently generate a random variable X and test the correlation between Y and X, we know that the histogram of the p-values of the correlation test should show a ...
0
votes
0answers
36 views

Finding β for binomial experiment

$X\sim \text{Binomial}(5,P)$ Rejection region $X ≤1$ $H_\text{null}: P=0.6$ $H_\text{alt}: P<0.6$ Find $β$ at $P= 0.3$ I understand that ($β$ = not in the rejection region/H alt....
3
votes
0answers
49 views

How to determine the optimal ratio of a type I and type II error in a given business context? [closed]

Let us assume I have developed two designs (A, B) of a product and I want to see which one performs better in terms of sales. So I run an experiment where some customers can buy A whilst others can ...
0
votes
1answer
35 views

Using Benjamini and Hochberg's false discovery rate with one independent factor and multiple outcome variables

In a study where I compare the male vs. female university students (independent variable = gender) on 20 different outcomes (each referring to a wellbeing measure), can I apply a false discovery rate (...
1
vote
1answer
63 views

Calculate power of t test?

I was wondering how you would calculate the power of a one sided and two sided t test and whether it differed from the z test. I know for a z test it is $p(z < z_{alpha} - abs(u_{a} - u_0)/(s/sqrt(...
1
vote
0answers
43 views

Bayes Factor for model and variable selection and type I & II error rate

I am trying to implement Bayes Factor for model and variable selection in Bayesian Linear Regression and finding out corresponding type I and type II error. I need your help regarding this. I will be ...
0
votes
0answers
58 views

Can type 1 error give lower bound on power of test, if null-hypothesis is false

So basically I have the following scenario: I have developed a new way for conditional independence testing, and am bench-marking is against other methods. To do this, I test the power in scenarios, ...
1
vote
0answers
41 views

How to explain the difference in significance level and type I error in permutation test of Pearson correlation on non-normal bivariate data

Let $x,y$ be sampled from a bivariate distribution $\mathcal{D}$ with zero covariance / no correlation. Let $r(x,y)$ be the Pearson correlation between $x$ and $y$. Let $H_0$ be that there is no ...
2
votes
1answer
76 views

How to report findings to show lack of correlation?

I am looking at the relationship between two variables. Contrary to custom my given h0 is "there is a significant correlation" and my h1 is "there is no significant correlation". I calculated a ...
0
votes
1answer
58 views

Testing when the Null Hypothesis is that the distributions do differ. Application in the case of Kolmogorov-Smirnov test

My area of interest is Classification (Machine Learning). A Classifier will typically output a conditional probability distribution (P(Y=1|X) ) and the question may arise whether this predicted PDF ...
2
votes
1answer
154 views

Are type I error & FWER both conditional probabilities?

According to my textbook, type I error is defined as the conditional probability of rejecting $H_0$ given $H_0$ is correct. On the other hand, when I study the FWER (although the definition is related ...
0
votes
1answer
38 views

Probability that a classifier is consistent with n random, independent training examples?

Suppose that a classifier is bad if its true error rate is greater than $ \epsilon $. What is the probability that a bad classifier will be consistent with n random, independent training examples? ...
1
vote
1answer
46 views

Formula for calculating power in hypothesis testing

We calculate power in hypothesis testing as 1-beta (probability that null hypothesis is false given that alternative hypothesis is true). As this probability increases, power also increases. By ...
0
votes
0answers
21 views

Alpha and known Type I error rate

I think I miss some very basic concepts. Suppose we have a test and a true Type I error has been estimated under a particular alpha used to test significance. Suppose now this true error rate turned ...
1
vote
1answer
215 views

Hypothesis finding type 1 error probability

The manufacturer of bags of cement claims that they fill each bag with at least 50.1 pounds of cement. Assume that the standard deviation for the amount in each bag is 1.2 pounds. The decision rule is ...