Are type I error rates (alpha's) reliable?

Question

In a journal article, there can be dozens of statistical tests that are unrelated, and, the more tests there are, the greater the likelihood that the probabilities for one of those tests for a given type I error would convert from $p \lesssim0.05$ to $p \gtrsim0.05$ (or the obverse) with a different initial data set. Yet, no one has ever suggested doing a Bonferroni correction on an entire suite of unrelated statistical tests, just because a journal article has a larger or smaller number of statistical tests in its text. This is because it is a non-issue as journal articles are not statistical units (apples and oranges). The strategy I use for this is to regard probabilities with a somewhat jaundiced eye, that is, I consider probabilities that are $0.02<p<0.1$ or $0.01<p<0.2$, depending on the context as borderline, and not strongly indicative of anything. I would be interested to hear what my colleagues on this site think of that approach. I am most interested in hearing what my colleagues think about probabilities in the neighborhood of $0.05$, which I do not think is reliable enough to be strongly indicative of anything, that is, without knowing also how variable that probability would be in a repeat experiment.

Also see https://stats.stackexchange.com/a/33825/99274 which gives an example for $\alpha=0.05$. So, shouldn't we be examining reliability for probabilities? This is also relevant to current concerns, e.g., Is this the solution to the p-value problem?

Answer 1

Type II errors are rarely if ever quantified. Usually, you select, among the available tests which don't inflate type I error, the one that has the highest power (and thus the lowest type II error ceteris paribus).

It is difficult to quantify type II errors because it depends on the (unknown, otherwise there would be no need to test anything at all) true population average. You can say for instance:

If the true population average is x, then by comparing it to our null hypothesis value of 0, we have a probability of y to commit type II error.

True population averages that are distinct from-, but very close to the value under the null hypothesis have much higher probabilities of type II errors than those that are far off the value under the null hypothesis. (Arguably, those type II errors, though more likely, are also less severe since the size of the effect that we missed to detect is smaller)

Answer 2

One may be able to bootstrap the data and find the envelope (distribution) of probabilities resulting from serial application of a test of hypothesis, and after inspection of that distribution find an appropriate estimate of location and dispersion of probability.

OK, so much for confusing verbiage. Let's do some Monte Carlo simulations to see what this means. Let us generate t-tests in three groups to test for the probability of two normal distributions having different means. The distributions are $\mathcal{N}(1,1)$ and $\mathcal{N}(2,1)$, i.e., the means are different, 1, and 2, and the standard deviations are the same, 1.

• Group (1) Each distribution has 10 random normal realizations, repeated 10,000 times for 10,000 t-tests.
• Group (2) Each distribution has 20 random normal realizations, repeated 5,000 times for 5,000 t-tests.
• Group (3) Each distribution has 40 random normal realizations, repeated 2,500 times for 2,500 t-tests

Results: Group (1) Percentage p-values below 0.05 is 55.73%

Results: Group (2) Percentage p-values below 0.05 is 86.52%

Results: Group (3) Percentage p-values below 0.05 is 99.2%

Discussion: From the simulations, it is obvious that Group (1) results are not reliable, and Group (2) results better but could still be misleading. Only the Group (3) results inspire some sense of confidence that the p-values mean something. Now, if instead of $\mathcal{N}(1,1)$ and $\mathcal{N}(2,1)$, we had chosen $\mathcal{N}(1,1)$ and $\mathcal{N}(10,1)$, the Group (1) results would have been much more reliable. Group (3) has another interesting property, that is, we can attempt to balance errors, that is, percentage p-values below 0.02 is 98.16%. That means a lot more than stating the $\alpha=0.05$ and ignoring if that is appropriate or not. If we tried to do something similar with Group (2) we would quickly come to the conclusion that it is not significant enough of a test, i.e, for Group (2) the percentage of p-values below 0.1 is 92.2%

There is nothing preventing us from doing tests like this with actual data, for example with bootstrap resampling. The answer here is that without testing, we don't know what the meaning of $p=0.01$ is if it could have easily been $p=0.1$, and without some idea of how reliable a probability value is we just don't know.