# Are type I error rates (alpha's) reliable?

In a journal article, there can be dozens of statistical tests that are unrelated, and, the more tests there are, the greater the likelihood that the probabilities for one of those tests for a given type I error would convert from $$p \lesssim0.05$$ to $$p \gtrsim0.05$$ (or the obverse) with a different initial data set. Yet, no one has ever suggested doing a Bonferroni correction on an entire suite of unrelated statistical tests, just because a journal article has a larger or smaller number of statistical tests in its text. This is because it is a non-issue as journal articles are not statistical units (apples and oranges). The strategy I use for this is to regard probabilities with a somewhat jaundiced eye, that is, I consider probabilities that are $$0.02 or $$0.01, depending on the context as borderline, and not strongly indicative of anything. I would be interested to hear what my colleagues on this site think of that approach. I am most interested in hearing what my colleagues think about probabilities in the neighborhood of $$0.05$$, which I do not think is reliable enough to be strongly indicative of anything, that is, without knowing also how variable that probability would be in a repeat experiment.

Also see https://stats.stackexchange.com/a/33825/99274 which gives an example for $$\alpha=0.05$$. So, shouldn't we be examining reliability for probabilities? This is also relevant to current concerns, e.g., Is this the solution to the p-value problem?

• Comments are not for extended discussion; this conversation has been moved to chat. – whuber Nov 3 '16 at 16:22

Type II errors are rarely if ever quantified. Usually, you select, among the available tests which don't inflate type I error, the one that has the highest power (and thus the lowest type II error ceteris paribus).

It is difficult to quantify type II errors because it depends on the (unknown, otherwise there would be no need to test anything at all) true population average. You can say for instance:

If the true population average is x, then by comparing it to our null hypothesis value of 0, we have a probability of y to commit type II error.

True population averages that are distinct from-, but very close to the value under the null hypothesis have much higher probabilities of type II errors than those that are far off the value under the null hypothesis. (Arguably, those type II errors, though more likely, are also less severe since the size of the effect that we missed to detect is smaller)

• We usually assume type I to be 0.05, then compute type II and compare it to 0.05. I do not understand why you say that type II errors are rarely quantified. Do not most tests do exactly that? – Carl Oct 29 '16 at 17:56
• If I didn't know better I would say you are confusing type I with type II. – David Ernst Oct 30 '16 at 18:25
• Probably, you are implying a specific effect size for which you can define a bound on beta. beta is only defined in relation to that effect size, not in general. Also, this may differ from field to field, but I have only seen papers that set a cutoff for alpha, select the most powerful available test and do not quantify beta. – David Ernst Oct 30 '16 at 18:40
• Right you are, I messed up. Maybe change answer not that question is changed. – Carl Nov 2 '16 at 1:57
• Changed the answer, and gave an example using Monte Carlo simulation. Didn't show bootstrap for actual data, but the principle is the same. – Carl Dec 28 '20 at 1:14

One may be able to bootstrap the data and find the envelope (distribution) of probabilities resulting from serial application of a test of hypothesis, and after inspection of that distribution find an appropriate estimate of location and dispersion of probability.

OK, so much for confusing verbiage. Let's do some Monte Carlo simulations to see what this means. Let us generate t-tests in three groups to test for the probability of two normal distributions having different means. The distributions are $$\mathcal{N}(1,1)$$ and $$\mathcal{N}(2,1)$$, i.e., the means are different, 1, and 2, and the standard deviations are the same, 1.

• Group (1) Each distribution has 10 random normal realizations, repeated 10,000 times for 10,000 t-tests.
• Group (2) Each distribution has 20 random normal realizations, repeated 5,000 times for 5,000 t-tests.
• Group (3) Each distribution has 40 random normal realizations, repeated 2,500 times for 2,500 t-tests

Results: Group (1) Percentage p-values below 0.05 is 55.73%

Results: Group (2) Percentage p-values below 0.05 is 86.52%

Results: Group (3) Percentage p-values below 0.05 is 99.2%

Discussion: From the simulations, it is obvious that Group (1) results are not reliable, and Group (2) results better but could still be misleading. Only the Group (3) results inspire some sense of confidence that the p-values mean something. Now, if instead of $$\mathcal{N}(1,1)$$ and $$\mathcal{N}(2,1)$$, we had chosen $$\mathcal{N}(1,1)$$ and $$\mathcal{N}(10,1)$$, the Group (1) results would have been much more reliable. Group (3) has another interesting property, that is, we can attempt to balance errors, that is, percentage p-values below 0.02 is 98.16%. That means a lot more than stating the $$\alpha=0.05$$ and ignoring if that is appropriate or not. If we tried to do something similar with Group (2) we would quickly come to the conclusion that it is not significant enough of a test, i.e, for Group (2) the percentage of p-values below 0.1 is 92.2%

There is nothing preventing us from doing tests like this with actual data, for example with bootstrap resampling. The answer here is that without testing, we don't know what the meaning of $$p=0.01$$ is if it could have easily been $$p=0.1$$, and without some idea of how reliable a probability value is we just don't know.

• when you say ''find the envelope (distribution) of probabilities '', what do you mean by probability ? The p-value ? – user83346 Nov 3 '16 at 6:58
• @fcop Yes. The standard abbreviation for probability is $p$. – Carl Nov 3 '16 at 14:04
• And what is then ''a probability resulting from a hypothesis test'' ? – user83346 Nov 3 '16 at 15:18
• @fcop onlinecourses.science.psu.edu/statprogram/node/138 Why are you constantly asking the same thing? Is there some purpose to this? – Carl Nov 3 '16 at 15:22
• I just say that a p value and the probability of a type I error are different things – user83346 Nov 3 '16 at 15:41