I apologize ahead of time if this a trivial question. I am building probability models for games - and in order to test the game code, I am comparing the results of large iterations of games with the original probability models. Specifically, I am calculating Standard Deviation, Skewness and Excess Kurtosis and using these calculations for verification. The sticking point lies in calculating appropriate confidence intervals since the underlying distributions are almost always not normal. I have been searching for quite some time for a solution - and the only thing which seems apparent is the BCa bootstrapping method, though I am not sure that this is really appropriate or necessary. It seems to me that there ought to be a way, given the original distribution, to properly calculate the expected variation not only of the mean, but also of the standard deviation, skewness and kurtosis. Is this possible?
- Due to some helpful comments, I am going to try to rephrase this (you may ignore the comments that I tried to make in response as they are a bit disjointed):
Okay, let me see if I can rephrase this thought taking away the focus of the games since that is only relevant in terms of context. If I have a discrete distribution which is not normal - and then I proceed to recreate this distribution through some sort of Monte Carlo process - what is the best way to check that the Monte Carlo process is accurately recreating the original distribution? My thought was that since a given distribution is characterized by its moments, I could use this characterization. The problem with this approach is how to estimate whether the observed error in the moments (from the Monte Carlo simulation) is significant. It seems to me that there should be some way to do this.