I have been using Monte Carlo simulation to evaluate A/B testing algorithms for quite a while. I noticed some simulation cases produced by simulation are never seen in real data. For example, the attached graph is p-value chart for testing if version 1 is better than version 0. You can notice p-value reaches above 95% at around sample 25,000 and then quickly drops below 5% around sample 50,000. I never saw this kind of pattern in real data. If this is a valid observation, how to explain it? Is it because real data is somehow auto-correlated while simulation is conducted under the assumption that observations are generated independently(i.i.d)?
There may be an issue on the cycle length of the random number generator, which could be an issue for large n!
Apparently, cycles repeat at some point.
Try adding a random choice to which of two (or more) possible new random #s you will select. If a repeating cycle issue, this will add randomization.
See if you can detect any improvement.