Experiment or simulation to undestand type I and type II errors I am studying Type I and Type II errors and basic concepts of testing hypotheses. To better develop my intuition I would like to write a few simple simulations.
What I'm hoping is to get a base simulation going, that I can then tweak to understand this area.
I am not looking for code in a specific language. However, pseudo code would be great, or references to a well documented example.
Thanks.
 A: This would be the most basic procedure behind any such simulation:  
Type I errors: 


*

*Have the computer generate a set (of size $n$) of pseudorandom numbers that conform to a particular distribution (the normal would be most typical).  

*Generate a second identical set (i.e., same distribution, parameters, and size).  

*Conduct a statistical test on these data (as I have described this, a t-test would be appropriate).  

*Store the resulting p-value.  

*Iterate over (repeat) the above procedure many times (e.g., 10k is popular).  

*Determine the proportion of observed p-values fall below your chosen $\alpha$ level (typically .05).
(Note that this observed proportion ought to be very close to $\alpha$.)  


Type II errors (modify the above procedure as follows):   
For the second step: Generate a second set of pseudorandom numbers that differ from the first set in a pre-specified way (typically the mean would differ by some amount).  
On the sixth step: the proportion of observed p-values below $\alpha$ will almost certainly differ from $\alpha$ by a large amount.  The observed proportion is an estimate of the statistical power of your test for that exact situation (i.e., data from those distributions, with those parameters, with those $n$'s).  

Using simulations in a manner like this to explore properties of tests or situations, or to conduct power analyses is very common.  Moreover, they have been commonly used on this site to demonstrate / explain statistical concepts.  Here are some threads you can explore if you want:  


*

*Here is an example where I used such a simulation to show that power decreases as the two group sizes become increasingly unequal: How should one interpret the comparison of means from different sample sizes?  

*This is a very extensive discussion of using simulations to assess power: Simulation of logistic regression power analysis - designed experiments.  

*A broader discussion of using simulations to understand statistical concepts can be found here: Using computer simulations to better understand statistical concepts at the graduate level.  

A: The most common example is of this kind: take a normal variate, $X_1$, which can be either $\mathcal{N}(0,1)$ or $\mathcal{N}(2,1)$. If you build a test accepting $\mathcal{N}(0,1)$ when $x_1<1.68$ and rejecting $\mathcal{N}(0,1)$ when $x_1>1.68$, it is rather simple to check by simulation that the type I error is $0.05$ and the type II error is $0.37$. For instance,
#type I error
x=rnorm(10^3)
sum(x>1.68)/10^3

and
#type II error
x=rnorm(10^3,2)
sum(x<1.68)/10^3

Of course, simulation is not very helpful in this case, where everything can be computed analytically.
A: Check out Geoff Cumming's "dancing p-values"
http://www.youtube.com/watch?v=ez4DgdurRPg&feature=plcp
Cummings is the author of "Understanding the new statistics."
