interdependence of type 1 error and type 2 error in p-Value based hypothesis tests

Question

Investigating a t-Test, I ran some "experiments", generating randomly distributed values around a given mean, and running a whole bunch of t-test (always with new data) for conditions where the null-hypothesis is true, and I indeed found, that the type 1 error, the ratio of the number of times the t-test incorrectly rejected the null-hypothesis and the number of total times, $\alpha$, I ran: $\alpha$ matched the p-Value.

Then I changed the script, choosing a different mean for the random-number generator, so I set up the experiment in a way that the alternative hypothesis must be true. I then calculated the empirical rate for a type II error $\beta$.

I then varied the p-Value that I used in the t-test and realized that the lower I chose the p-Value, the higher my type II error got.

I understand the meaning of both types of errors individually (I believe), but I have somewhat of a hard time reasoning about their interdependence.

• Is this a general property of type 1 and type 2 errors?
• Is it specific to the t-Test?
• is there a good way to quantify this?

Answer 1

To summarize, you find that if you use a lower p-value as threshold to reject hypotheses, the type I error goes down and the type II error goes up.

This should make sense; if you use a lower threshold (in terms of p-values) for rejecting a null hypothesis, you will be rejecting fewer hypotheses. For one, that will make it less likely that you will falsely reject (reject even though the null is true) as you are being more conservative. Hence, the Type I error should go down (indeed it should be exactly the threshold you use for the p-value).

On the other hand, since you are being more conservative in rejecting the null hypothesis, you are also more likely to not reject when the null hypothesis is false. In a sense, you are requiring "more evidence" against the null hypothesis. That is, the Type II error rate goes up.

Maybe it is also good to consider the extremes: if you never reject the null hypothesis (p-value threshold of 0), then your type I error rate is 0. You never reject so you never make a mistake. On the other hand, your Type II error rate is 1.

On the other hand, if you always reject, then your Type II error rate is 0, because you never make a mistake not rejecting. But, your Type I error rate is 1 in this case.