2
$\begingroup$

I have been using Monte Carlo simulation to evaluate A/B testing algorithms for quite a while. I noticed some simulation cases produced by simulation are never seen in real data. For example, the attached graph is p-value chart for testing if version 1 is better than version 0. You can notice p-value reaches above 95% at around sample 25,000 and then quickly drops below 5% around sample 50,000. I never saw this kind of pattern in real data. If this is a valid observation, how to explain it? Is it because real data is somehow auto-correlated while simulation is conducted under the assumption that observations are generated independently(i.i.d)?

enter image description here

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ You need to tell us more on what is your real data and how did you conduct the Monte Carlo simulation, otherwise this is unanswerable. Obviously: if you ran MC simulation that makes wrong assumptions, than it would not give results that are aligned with the data. $\endgroup$
    – Tim ♦
    Jul 19, 2020 at 6:53

1 Answer 1

Reset to default
1
$\begingroup$

There may be an issue on the cycle length of the random number generator, which could be an issue for large n!

Apparently, cycles repeat at some point.

Try adding a random choice to which of two (or more) possible new random #s you will select. If a repeating cycle issue, this will add randomization.

See if you can detect any improvement.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Not sure I understand your answer. Can you elaborate on cycle repeat or offer links to articles/papers talking about this phenomena? $\endgroup$
    – etang
    Jul 19, 2020 at 0:33
  • 1
    $\begingroup$ Wit repeating cycles, you are assuming in here that they had generated more than $2^{19937} - 1$ samples? My best guess is that this was not the case. $\endgroup$
    – Tim ♦
    Jul 19, 2020 at 6:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.