I am studying Type I and Type II errors and basic concepts of testing hypotheses. To better develop my intuition I would like to write a few simple simulations.

What I'm hoping is to get a base simulation going, that I can then tweak to understand this area.

I am not looking for code in a specific language. However, pseudo code would be great, or references to a well documented example.



3 Answers 3


This would be the most basic procedure behind any such simulation:

Type I errors:

  1. Have the computer generate a set (of size $n$) of pseudorandom numbers that conform to a particular distribution (the normal would be most typical).
  2. Generate a second identical set (i.e., same distribution, parameters, and size).
  3. Conduct a statistical test on these data (as I have described this, a t-test would be appropriate).
  4. Store the resulting p-value.
  5. Iterate over (repeat) the above procedure many times (e.g., 10k is popular).
  6. Determine the proportion of observed p-values fall below your chosen $\alpha$ level (typically .05).
    (Note that this observed proportion ought to be very close to $\alpha$.)

Type II errors (modify the above procedure as follows):

For the second step: Generate a second set of pseudorandom numbers that differ from the first set in a pre-specified way (typically the mean would differ by some amount).

On the sixth step: the proportion of observed p-values below $\alpha$ will almost certainly differ from $\alpha$ by a large amount. The observed proportion is an estimate of the statistical power of your test for that exact situation (i.e., data from those distributions, with those parameters, with those $n$'s).

Using simulations in a manner like this to explore properties of tests or situations, or to conduct power analyses is very common. Moreover, they have been commonly used on this site to demonstrate / explain statistical concepts. Here are some threads you can explore if you want:


The most common example is of this kind: take a normal variate, $X_1$, which can be either $\mathcal{N}(0,1)$ or $\mathcal{N}(2,1)$. If you build a test accepting $\mathcal{N}(0,1)$ when $x_1<1.68$ and rejecting $\mathcal{N}(0,1)$ when $x_1>1.68$, it is rather simple to check by simulation that the type I error is $0.05$ and the type II error is $0.37$. For instance,

#type I error


#type II error

Of course, simulation is not very helpful in this case, where everything can be computed analytically.


Check out Geoff Cumming's "dancing p-values"


Cummings is the author of "Understanding the new statistics."


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.