Rejecting the null hypothesis when it is true.

learn more… | top users | synonyms

3
votes
2answers
190 views

Is p-value also the false discovery rate?

In http://surveyanalysis.org/wiki/Multiple_Comparisons_(Post_Hoc_Testing) It states "For example, if we have a p-value of 0.05 and we conclude it is significant the probability of a false discovery ...
0
votes
1answer
27 views

post-hoc test after logistic regression with interaction. Risk higher for type 1 or type 2 error?

I have a large data set (>1000 obs) and i'm performing regressions tests, both linear and logistic, on a series of clinical outcomes. In this test I verify the effect of interactions between two cat ...
5
votes
1answer
157 views

Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same ...
0
votes
1answer
23 views

Is it possible to randomly sample from single data set (Monte Carlo style) to create new data sets?

Background I understand Monte Carlo methods only superficially, but I understand you can repeatedly randomly sample, with or without replacement, from your data set to estimate population parameters ...
0
votes
0answers
15 views

Finding the critical region which minimizes the type 2 error

Kinda have another question about hypothesis testing. alpha is type 1 error while beta is type 2 error. I'm a bit confused in what I'm supposed to do. I attempted to answer the question but I ...
0
votes
0answers
16 views

Advice on how to analyze these data

In my experiment, I have 312 separate samples, each of which is made up of several different food items. I multiply each food by a specific multiplier to convert it to energy, and then sum the energy ...
1
vote
1answer
55 views

How to control for type-1-error inflation with multiple chi-squared tests?

I have two between-subject factors, each with two levels (so 4 conditions). Furthermore, I have one dependent variable (qualitative), also consisting of two levels. Now I want to make pairwise ...
0
votes
1answer
42 views

glm inflated error…why?

I'm pretty new to stats, so this may be dumb. I've been running a bunch of models on randomly generated data to try and develop my understanding of type 1 error. I've noticed that using ...
0
votes
1answer
26 views

Confidence Interval Coverage-error and Type I error

Could somebody explain to me the relationship between coverage error and type one errors in multiple comparisons testing, if there is one in fact? Does a coverage error occur when the true value of ...
1
vote
0answers
65 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...
0
votes
0answers
16 views

Identifying Type 1 errors using confidence intervals

I'm working with an SPSS data set, a Monte Carlo study with 5000 trials. There are 5 means and therefore 10 comparisons for each trial. I'm provided with the t-test significance of each comparison ...
2
votes
2answers
58 views

Comparing different conditions on a binomial distribution

I have some data where I have tested a binomial random variable under 4 conditions. The null hypothesis is that they all have equal means, alternative hypothesis is that one or more means differ from ...
1
vote
0answers
115 views

2x2x2 repeated measures anova: do I correct for type I error?

I have a 2x2x2 repeated measures design and am trying to decompose a 3-way interaction. As all my factors have 2 levels, I always reach the point that I am doing pairwise comparisons for the simple ...
6
votes
1answer
102 views

How to measure uncertainty of a parameter when false positives exist?

The main goal of my research is to measure the percentage of brown dwarf stars in the Pleiades star forming cluster that are actually double stars (i.e. the brown dwarf star has a companion brown ...
0
votes
0answers
79 views

Analyzing type I error rate in a simulation study using ANOVA-like methods

I've run a simulation study in order to determine type I error rate of a statistic.My simulation design includes threes factors as sample size (4 levels), test length or number of items (3 levels) and ...
1
vote
2answers
138 views

Three important issues

I don't understand three important issues : (1) Why do we calculate the test statistic under the null hypothesis ? (2) Why do we reject the null hypothesis when : p-value < significance level ? ...
0
votes
1answer
67 views

Can I make any tests statistically significant?

I am using a simple $\chi^2$ test to see if there is any impact of an independent variable (two charts) on the resulting choices (two choices). I found that as the number of sample increase, the ...
2
votes
2answers
127 views

Multiple resampling test/train dataset when choosing new models?

I have been reading several posts on testing multiple models on the same dataset, which can lead to problems controling type-1 errors. Mostly these posts have to do with data-mining on big datasets: ...
3
votes
1answer
230 views

Determining sample size with a proportion and binomial distribution

I am trying to learn some statistics using the book, Biometry by Sokal and Rohlf (3e). This is an exercise in the 5th chapter which covers probability, the binomial distribution, and Poisson ...
5
votes
1answer
146 views
1
vote
0answers
153 views

Calculating the size of Type 1 error, Type 2 error and power of the test

Let $X$ have a binomial distribution with parameter $n=5$ and $P\in [p:p=\frac{1}{4},\frac{1}{2}]$. The null hypothesis $H_{0}:P=\frac{1}{4}$ is rejected, and The alternative hypothesis ...
0
votes
0answers
22 views

Checking significance of all prefixes of a coin flip sequence

Derived from this blog post and similar to this question: Given a random walk generated from flipping a coin, what is the function for the envelope that a fair coin will not exit at any point (with ...
10
votes
1answer
482 views

How and when to use the Bonferroni adjustment

I have two questions regarding when to use a Bonferroni adjustment: Is it appropriate to use a Bonferroni adjustment in all cases of multiple testing? If one performs a test on a data set, then one ...
1
vote
1answer
175 views

Type 1 error Bayesian Stats

Is there anybody who can tell me or explain to me about finding the Type 1 error for a Bayesian clinical trial. Of a 100 patients I need 59 successful patients to exceed my posterior threshold. I ...
3
votes
2answers
851 views

Simple and paired t-tests and the control for multiple comparisons

Let's say that I have a group of subjects that were measured in two conditions (A and B) and I want to check three hypotheses: i) do data from A and B differ from each other?, ii) do data from A ...
3
votes
2answers
200 views

Rationale behind using ANOVA to compare means

Let's say we have 7 samples of data and we wish to test whether there is a significant difference between the means of each sample. We could use t-tests, which would require 21 different t-tests. ...
0
votes
0answers
46 views

What is the type 1 error of rejecting samples with low p-values?

Consider an amateur statistician who will discard any sample which does not lead him to reject his null hypothesis at an alpha=.05 significance level with a two-sided test. What is this statisticians ...
2
votes
1answer
768 views

Correcting for Type 1 error in multiple paired t-tests?

I'm wondering whether or not I should adjust the significance level of paired t-tests due to multiple tests (to avoid the possibility of Type 1 error), although the tests are independent. Here's what ...
3
votes
1answer
220 views

On FDA guidance about Bayesian practice

US FDA authorizes the use of Bayesian statistics with informative priors (in certain contexts): ...
2
votes
1answer
179 views

Is a MANOVA justified?

I have to analyze the data from a study in which two groups of subjects (total sample = 30) were tested on 4 cognitive tasks, all realtive to executive functions. For each task, I have accuracy score ...
1
vote
1answer
272 views

Determining number of tests to correct for using Bonferroni

I am doing final revisions to my dissertation, which includes a large number of regressions. I have six outcome variables (DVs) and I am testing the effect of four primary (correlated) IVs, but I am ...
13
votes
2answers
322 views

Optional stopping rules not in textbooks

Stopping rules affect the relationship between P-values and the error rates associated with decisions. A recent paper by Simmons et al. 2011 coins the term researcher degrees of freedom to describe a ...
2
votes
1answer
158 views

Controlling type I error in permutation tests

I'm implementing a permutation test for a test statistic with unknown distribution. Apart from my concrete situation (see below), I'd like to know first If and under what circumstances permutation ...
2
votes
2answers
65 views

Conditions under which the power curve is not smooth or continuous in Hypothesis testing

Not sure if this is a valid question. I was wondering if there is are good answers to this question: Under what conditions is the power curve not smooth or continuous in Hypothesis testing? ...
4
votes
3answers
616 views

Experiment or simulation to undestand type I and type II errors

I am studying Type I and Type II errors and basic concepts of testing hypotheses. To better develop my intuition I would like to write a few simple simulations. What I'm hoping is to get a base ...
1
vote
1answer
113 views

Prove this theorem related with specification tests

I'm not a mathematician and I need to prove what this theorem says. I think is easy and I know how it works, but definitively I'm not too much rigorous to make a demonstration. Can anybody help me? I ...
2
votes
1answer
254 views

The impact of non-normality (contaminated normal) on type 1 error and power when sigma is known

When sigma is unknown, contaminated normal distributions tend to have (1) the issue of the actual Type 1 error rate being lower than the nominal level and (2) the issue of reduced power. This pattern ...
3
votes
1answer
202 views

Overall type I error with dependent tests

Imagine we test 2 dependent samples: For example, we first test an overall sample (treatment vs. control) and then a subset of this overall sample (treatment sbg vs. control sbg). In this case, we are ...
2
votes
1answer
404 views

Frequentist properties of p-values in relation to type I error

This question is related to frequentist properties of p-values and their relation to type I error and why the results from an online simulation differ from what I would have expected. Assume that I ...
1
vote
0answers
57 views

Relativization and tackling an overdispersed dataset

If there is a species that is hyperabundant in a dataset and makes up 284/1638 individuals yet originated from just 1 of 10 traps. Is a quassi-poisson GLM sufficient providing residual deviance is ...
0
votes
1answer
517 views

What type of error correction do I have to use after Wilcoxon matched pairs test?

Do I have to use Bonferroni's or Holm's corrections? Please see this previous question for my example data and the design. (Shortly, I have three or more dependent groups.)
6
votes
2answers
3k views

How to test whether a regression coefficient is moderated by a grouping variable?

I have a regression done on two groups of the sample based on a moderating variable (say gender). I'm doing a simple test for the moderating effect by checking whether the significance of the ...
11
votes
1answer
769 views

Overall type I error when repeatedly testing accumulating data

I have a question about group sequential methods. According to Wikipedia: In a randomized trial with two treatment groups, classical group sequential testing is used in the following manner: If n ...
38
votes
23answers
11k views

Is there a way to remember the definitions of Type I and Type II Errors?

I'm not a statistician by education, I'm a software engineer. Yet statistics comes up a lot. In fact, questions specifically about Type I and Type II error are coming up a lot in the course of my ...