# Why Some Simulation Scenarios Don't Look Real? [closed]

I have been using Monte Carlo simulation to evaluate A/B testing algorithms for quite a while. I noticed some simulation cases produced by simulation are never seen in real data. For example, the attached graph is p-value chart for testing if version 1 is better than version 0. You can notice p-value reaches above 95% at around sample 25,000 and then quickly drops below 5% around sample 50,000. I never saw this kind of pattern in real data. If this is a valid observation, how to explain it? Is it because real data is somehow auto-correlated while simulation is conducted under the assumption that observations are generated independently(i.i.d)?

• You need to tell us more on what is your real data and how did you conduct the Monte Carlo simulation, otherwise this is unanswerable. Obviously: if you ran MC simulation that makes wrong assumptions, than it would not give results that are aligned with the data.
– Tim
Jul 19, 2020 at 6:53

• Wit repeating cycles, you are assuming in here that they had generated more than $2^{19937} - 1$ samples? My best guess is that this was not the case.