Investigating a t-Test, I ran some "experiments", generating randomly distributed values around a given mean, and running a whole bunch of t-test (always with new data) for conditions where the null-hypothesis is true, and I indeed found, that the type 1 error, the ratio of the number of times the t-test incorrectly rejected the null-hypothesis and the number of total times, $\alpha$, I ran: $\alpha$ matched the p-Value.
Then I changed the script, choosing a different mean for the random-number generator, so I set up the experiment in a way that the alternative hypothesis must be true. I then calculated the empirical rate for a type II error $\beta$.
I then varied the p-Value that I used in the t-test and realized that the lower I chose the p-Value, the higher my type II error got.
I understand the meaning of both types of errors individually (I believe), but I have somewhat of a hard time reasoning about their interdependence.
- Is this a general property of type 1 and type 2 errors?
- Is it specific to the t-Test?
- is there a good way to quantify this?