I have a statistic that I know was found with a Monte Carlo simulation. I also have other data associated such as standard deviation and number of simulations run. I want to verify the statistic with another independently built simulator. How do I compare the two simulation runs in a way that I can be confident they are converging to the same value?

When I have a theoretical value, I can construct a confidence interval around my simulated statistic and see if the theoretical falls in that range, but with a simulated value, I'm not so sure it's good to ignore the variance.


You can perform an independent samples t-test on the simulation outcomes. The test statistic is a function of the means of the estimated statistics, their standard deviations, and the number of simulation replications, all information you have available to you. See here for the equations.

| cite | improve this answer | |

I present a supporting view of mine on random number generators, to quote a source:

“One thing that traditional computer systems aren’t good at is coin flipping,” says Steve Ward, Professor of Computer Science and Engineering at MIT’s Computer Science and Artificial Intelligence Laboratory. “They’re deterministic, which means that if you ask the same question you’ll get the same answer every time. In fact, such machines are specifically and carefully programmed to eliminate randomness in results. They do this by following rules and relying on algorithms when they compute.”

You can program a machine to generate what can be called “random” numbers, but the machine is always at the mercy of its programming. “On a completely deterministic machine you can’t generate anything you could really call a random sequence of numbers,” says Ward, “because the machine is following the same algorithm to generate them. Typically, that means it starts with a common ‘seed’ number and then follows a pattern.” The results may be sufficiently complex to make the pattern difficult to identify, but because it is ruled by a carefully defined and consistently repeated algorithm, the numbers it produces are not truly random. “They are what we call ‘pseudo-random’ numbers,” Ward says.

For most applications, a pseudo-random number is sufficient, he adds. “For example, if you want to do a random sampling of a large set of data, you’ll need numbers to feed into the program so that the samples are more or less evenly distributed. Using pseudo-random numbers is perfectly acceptable in this case because there’s no quantitative advantage in the degree of randomness.” Similarly, a CD player in “random” mode is probably really playing in pseudo-random mode, with a pattern that is discernible if you listen carefully enough.

Not all randomness is pseudo, however, says Ward. There are ways that machines can generate truly random numbers. "

I you want to perform some comparative testing, here are a million digits of pi, which are truly random at https://www.piday.org/million/ .

[EDIT] Here is a source for my last suggestion on using the digits of pi, to quote:

Numbers like pi are also thought to be "normal," which means that their digits are random in a certain statistical sense. ... In the first six billion decimal places of pi, each of the digits from 0 through 9 shows up about six hundred million times.

So, to be clear, test each of your prior sources for 'random' digits against a series of numbers from pi for comparative tests (more than one kind of comparison property than just the frequency of Os and 1s, use Chi-squared test or even Wald–Wolfowitz runs test, see discussion here).

| cite | improve this answer | |
  • $\begingroup$ This doesn't answer the question at all. $\endgroup$ – Noah Apr 15 at 5:01
  • $\begingroup$ Thanks, for the comment, I have added a reference and made clear may implied testing process, compare each reputedly random simulation to results obtained with pi, truly random, as a benchmark. $\endgroup$ – AJKOER Apr 15 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.