1
$\begingroup$

I apologize ahead of time if this a trivial question. I am building probability models for games - and in order to test the game code, I am comparing the results of large iterations of games with the original probability models. Specifically, I am calculating Standard Deviation, Skewness and Excess Kurtosis and using these calculations for verification. The sticking point lies in calculating appropriate confidence intervals since the underlying distributions are almost always not normal. I have been searching for quite some time for a solution - and the only thing which seems apparent is the BCa bootstrapping method, though I am not sure that this is really appropriate or necessary. It seems to me that there ought to be a way, given the original distribution, to properly calculate the expected variation not only of the mean, but also of the standard deviation, skewness and kurtosis. Is this possible?

  • Due to some helpful comments, I am going to try to rephrase this (you may ignore the comments that I tried to make in response as they are a bit disjointed):

Okay, let me see if I can rephrase this thought taking away the focus of the games since that is only relevant in terms of context. If I have a discrete distribution which is not normal - and then I proceed to recreate this distribution through some sort of Monte Carlo process - what is the best way to check that the Monte Carlo process is accurately recreating the original distribution? My thought was that since a given distribution is characterized by its moments, I could use this characterization. The problem with this approach is how to estimate whether the observed error in the moments (from the Monte Carlo simulation) is significant. It seems to me that there should be some way to do this.

$\endgroup$
  • $\begingroup$ Hi there, do you mean that you are looking at the distributions of final scores for games? I'm not following what you mean by results of large iterations of games. $\endgroup$ – Michelle Feb 16 '12 at 9:23
  • $\begingroup$ You don't often see skewness and kurtosis used for this purpose because their standard errors are proportional to the sixth and eighth central moments of the distribution, respectively, and those can get huge. A question that describes what you want to know and the information you have (or can readily obtain), rather than focusing on a single technique you have adopted, would broaden the discussion and perhaps get you some more helpful replies. $\endgroup$ – whuber Feb 16 '12 at 15:38
1
$\begingroup$

If you have a discrete distribution, you would want to use Pearson $\chi^2$ test: $$ X^2 = \sum_{k=1}^K \frac{(n_k-np_k)^2}{p_k} $$ where $p_k$ is the known population probability, and $n_k$ is the sample frequency. Frankly, I doubt you will get much value of this test, though, as with a single testing, you will never know if you have genuine differences, or just type I error. What you could do though is to break your series into shorter chunks, compute either the $X^2$ or $p$-values, and test them against the target distribution ($\chi^2_{K-1}$ for the test statistic, $U[0,1]$ for the $p$-values) using Kolmogorov-Smirnov test.

I totally agree with whuber that working with moments is pointless. Moreover, for high skewness and kurtosis, the sample estimates may not be informative: there are upper limits on these sample statistics, so a kurtosis of say 10000 will never materialize in a sample of $n=50$ observations. See Cox (2010). This is a mathematical curiosity that I have never seen in my graduate coursework, although I now think it is an important caution against using the sample moments indiscriminately.

$\endgroup$
  • $\begingroup$ I have thought about the X^2 test, but as you noted am still grappling with the issue of error. The thing is, I am using sample sizes in the neighborhood of 10^10 (that is, 10^10 iterations of a Monte Carlo type sim), so the Standard Deviation, Skewness, etc., are all very close given that the sim is accurate (I've tested this). What I am looking for - and maybe there is not a good method, though I find this hard to believe - is a way to measure expected variation in, for example, skewness. $\endgroup$ – cyrus1.618 Feb 16 '12 at 19:05
  • $\begingroup$ As @whuber explained, variability in skewness depends on the 6th order moments. With some moderate paper-and-pencil work, you should be able to derive the variance of the sample third moment (unstandardized, without dividing by the standard deviation) to construct an asymptotic $z$-test. Derivations of this kind have been done to test normality (en.wikipedia.org/wiki/Jarque-Bera_test). $\endgroup$ – StasK Feb 17 '12 at 16:19
  • $\begingroup$ Awesome, thanks for highlighting this. I have looked into the Kolmogorov-Smirnov test, but it seems to me that since the target distribution is not continuous that this might be problematic. What are your thoughts on this? $\endgroup$ – cyrus1.618 Feb 17 '12 at 20:28
  • $\begingroup$ According the wikipedia article which you reference, "the Jarque-Bera test is a goodness-of-fit test of whether sample data have the skewness and kurtosis matching a normal distribution". I can safely say that none of the distributions that I work with are normal. How does this test help me? Thanks in advance. $\endgroup$ – cyrus1.618 Feb 17 '12 at 23:42
  • $\begingroup$ I'm sorry, disregard the above comment, I see that you were making an example. Thanks $\endgroup$ – cyrus1.618 Feb 17 '12 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.