1
$\begingroup$

Analyses based on iterative model fitting or simulations are typically seeded with an unpredictable random value based on a PRNG function plus the program / computer state. This causes analyses to return different results each time, unless the user remembers to hard-code a random seed into their analysis. It can lead to problems when comparing different runs, reproducing analyses that have been reported in publications, or dealing with potential for p-hacking in cases of borderline significance.

Since statistical analyses are meant to be precisely replicable, why isn't the default behavior to use a deterministic seed? For example, a seed could be based on a hash of the dataset or the regularlized command input. This would cause replications (identical analysis with identical data in the same software) to return reliable results, unless the user intentionally specifies a different seed.

The current default does ensure that if a seed leads to pathological model results (e.g., a terrible local maximum), the next run won't have the same problem. But statisticians usually shouldn't be re-running the same command to see whether it turns out differently anyway (outside of methods like bootstrapping, where ensembling is built in to the assumptions and the analysis). Is there another reason for using unpredictable and unrecorded random seeds?

$\endgroup$
5
  • 1
    $\begingroup$ I'm voting to close this question as off-topic because it is about how software designers make decisions and academics report results, not statistics as described in the help center. $\endgroup$ – Sycorax Feb 15 '17 at 18:22
  • 1
    $\begingroup$ One fairly obvious reason is that if the seed (and state) is always the same, so are the results. I'd be surprised to discover that a dice-rolling simulation always returns 1,4,3,2,5,1,6 as the first results each and every time I run it because that's not how randomness works. $\endgroup$ – Sycorax Feb 15 '17 at 18:24
  • 1
    $\begingroup$ @Sycorax: could you make your statement more precise (software, calibration, &tc.) ? $\endgroup$ – Xi'an Feb 15 '17 at 18:56
  • $\begingroup$ @Sycorax That's true, and I would expect that a function that produces random numbers as output would use a standard, unpredictable random seed. But I would expect a function that produces a parameter estimate or test statistic to produce the same results each time, and would be surprised if it didn't. $\endgroup$ – octern Feb 15 '17 at 21:08
  • 1
    $\begingroup$ @octern ...but those are also random numbers in the sense that they are subject to variation due to randomness. There's no distinction. $\endgroup$ – Sycorax Feb 15 '17 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.