1
$\begingroup$

Say I want to simulate from a density $f$.

I come up with two procedures for doing so, $proc1$ and $proc2$.

Now, I can obviously plot the histograms that come from simulating from $proc1$ and $proc2$ and plot $f$ on top of them, and say "yeah looks like they both work".

But what more can I say? Preferably, I want to be able to say something about the accuracy of those procedures. I want to be able to compare them and say something about which one is better or worse.

What can typically be done in this situation?

Right now, my only idea is to calculate the sample mean from both $proc1$ and $proc2$, and calculate its standard error, and then the simulation with a smaller error is better?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ What would you like to measure as “accuracy”? Set aside for now how to measure and assess it; that’s what we’ll help you do. But what do you want to measure to determine accuracy? Two examples are discrepancies between the simulated PDFs and the population PDFs and discrepancies between the simulated CDFs and population CDFs. $\endgroup$
    – Dave
    Commented Sep 22, 2019 at 0:11
  • $\begingroup$ Hi. I am not sure exactly, but let me give you an example. Say you want to use proc1 and proc2 to calculate the mean of $f$. So you sample 10.000 samples from proc1 and you take the average, and you do the same for proc2. Now, the "better/more accurate procedure" should be the one whose sample mean has a smaller standard error, right? So can those standard errors be used to compare accuracy? $\endgroup$
    – efeoeo
    Commented Sep 22, 2019 at 0:16
  • $\begingroup$ Why not just calculate the mean of $f$ the usual way? $\endgroup$
    – Dave
    Commented Sep 22, 2019 at 0:20
  • 1
    $\begingroup$ calculating the mean isn't the point, testing the simulation procedures is. $\endgroup$
    – efeoeo
    Commented Sep 22, 2019 at 0:22
  • $\begingroup$ The trouble with just assessing which procedure gives the lowest variance is that either procedure could be biased. I do not think that would be a good method. For determining which procedure is more accurate. My suggestion is to do something with the PDF or CDF. Do you have an equation for the PDF or CDF of $f$? $\endgroup$
    – Dave
    Commented Sep 22, 2019 at 0:30

1 Answer 1

Reset to default
1
$\begingroup$

Depending on your needs, some characteristics per sample ($\times 10^n$ as appropriate) of simulated $f$ (plus the CDF, $F=\int f$, and quantile function $F^{-1}$) that may be of interest for procedure 1 vs procedure 2:

  • Computational time (e.g., measured using clock cycles, seconds, or represented analytically using big O notation)

  • Memory cost

  • Energy cost

  • Uniformity of $p$ (from $F^{-1}$)

  • Accuracy in the tails of $f$

  • Accuracy for the distribution overall from an analytic formulation of $f$ (e.g., using a one-sample Kolmogorov-Smirnof test for each procedure for many samples of size $N$ and comparing the mean $p$ values).

The particular statistics tracked from simulating a univariate distribution will necessarily depend on the purposes envisioned for using it. For example, if the number of needed simulations is anticipated to be on the order of, say, $10^6$ per day or less, then you may not need to calculate compute time, memory, or energy costs. However, if anticipated need for simulated draws from $f$ is at a much greater rate, say $10^9$ per hour or more, then those costs per sample may be relevant.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ +1 I hadn’t even thought to consider the computational aspects. Just what do you mean by uniformity of p? $\endgroup$
    – Dave
    Commented Sep 22, 2019 at 0:13
  • $\begingroup$ @Dave I was imagining that deviations of $p$ from true uniform(0,1) imply some kind of bias from the target distribution, but haven't thought that through all the way. :) $\endgroup$
    – Alexis
    Commented Sep 22, 2019 at 0:15
  • $\begingroup$ You mean hitting the simulated distributions with the population CDF for a probability integral transform? $\endgroup$
    – Dave
    Commented Sep 22, 2019 at 0:16
  • $\begingroup$ @Dave depending on the nature of $f$, $F$, and $F^{-1}$ (any of which may or may not have a closed-form expression, and may necessarily entail approximation), procedure 1 and procedure 2 may be specific to one of these functions (standing on its own, not merely a probability transform of a simulation of one of the others). If I recall correctly, simulating the non-central $t$ distributions $f$ and $F$ are separate simulation algorithms, not simply probability transformations, for example. $\endgroup$
    – Alexis
    Commented Sep 22, 2019 at 0:18
  • $\begingroup$ @Dave I hope I understood your question. (And am not making too bad of a fool of myself in these comments. ;) $\endgroup$
    – Alexis
    Commented Sep 22, 2019 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.