# Analyzing type I error rate in a simulation study using ANOVA-like methods

I've run a simulation study in order to determine type I error rate of a statistic.My simulation design includes threes factors as sample size (4 levels), test length or number of items (3 levels) and estimator (3 levels). The statistic is developed to measure person fit with test data in educational testing situation.I've replicated the analysis in each cell (i.e. the design is fully-crossed) 100 times.

Now, I have the results and type I error rates range from 0.005 to 0.105 (i.e. across the whole analysis). I want to analyze how factors affect type I error rate using something similar to ANOVA. I tried Beta Regression in R using betareg package but I received this error message:

invalid dependent variable, all observations must be in (0, 1)

Any idea on how to determine the effect of design factors on type I error rate?

• Maybe post the code? – tchakravarty Jan 19 '14 at 8:19
• @fgnu this is the code I used m<-betareg(lz~size*length,data=df) in which size and length are factors related to sample size and test length.I couldn't do anything more. – Amin Jan 19 '14 at 8:30
• The code that you used to simulate the model, since based on that it should be possible to tell if you are violating the assumptions of the Beta regression model, as the function betareg is complaining. – tchakravarty Jan 19 '14 at 8:31
• @fgnu it's a long code.The statistic is called lz and theoretically follow a standard normal distribution. Then lz values below -1.65 are coded as 1 indicating a misfitting and 0 as fitting response pattern. I took the mean of 1s & 0s as type I error rate since the data were simulated based on the null hypothesis model.A sample of code can be found here tigger.uic.edu/~georgek/HomePage/… – Amin Jan 19 '14 at 8:36