# Experiment or simulation to undestand type I and type II errors

I am studying Type I and Type II errors and basic concepts of testing hypotheses. To better develop my intuition I would like to write a few simple simulations.

What I'm hoping is to get a base simulation going, that I can then tweak to understand this area.

I am not looking for code in a specific language. However, pseudo code would be great, or references to a well documented example.

Thanks.

This would be the most basic procedure behind any such simulation:

Type I errors:

1. Have the computer generate a set (of size $n$) of pseudorandom numbers that conform to a particular distribution (the normal would be most typical).
2. Generate a second identical set (i.e., same distribution, parameters, and size).
3. Conduct a statistical test on these data (as I have described this, a t-test would be appropriate).
4. Store the resulting p-value.
5. Iterate over (repeat) the above procedure many times (e.g., 10k is popular).
6. Determine the proportion of observed p-values fall below your chosen $\alpha$ level (typically .05).
(Note that this observed proportion ought to be very close to $\alpha$.)

Type II errors (modify the above procedure as follows):

For the second step: Generate a second set of pseudorandom numbers that differ from the first set in a pre-specified way (typically the mean would differ by some amount).

On the sixth step: the proportion of observed p-values below $\alpha$ will almost certainly differ from $\alpha$ by a large amount. The observed proportion is an estimate of the statistical power of your test for that exact situation (i.e., data from those distributions, with those parameters, with those $n$'s).

Using simulations in a manner like this to explore properties of tests or situations, or to conduct power analyses is very common. Moreover, they have been commonly used on this site to demonstrate / explain statistical concepts. Here are some threads you can explore if you want:

The most common example is of this kind: take a normal variate, $X_1$, which can be either $\mathcal{N}(0,1)$ or $\mathcal{N}(2,1)$. If you build a test accepting $\mathcal{N}(0,1)$ when $x_1<1.68$ and rejecting $\mathcal{N}(0,1)$ when $x_1>1.68$, it is rather simple to check by simulation that the type I error is $0.05$ and the type II error is $0.37$. For instance,

#type I error
x=rnorm(10^3)
sum(x>1.68)/10^3


and

#type II error
x=rnorm(10^3,2)
sum(x<1.68)/10^3


Of course, simulation is not very helpful in this case, where everything can be computed analytically.

Check out Geoff Cumming's "dancing p-values"