8
$\begingroup$

The spirit of this question comes from "Ordinary Monte Carlo", also known as "good old-fashioned Monte Carlo"

Suppose I have a random variable $X$, with
$$\mu := E[X]\\ \sigma^2:=Var[X] $$

Both are unknown values, because the probability distribution function of $X$ is unknown (or the computations are intractable).

Either way, suppose we can somehow simulate $n$ draws $X_1,X_2,\dots,X_n$ (these are independent and identically distributed) from the distribution of $X$. Let us define the sample parameters

$$ \hat{\mu}_n := \frac{1}{n}\sum_{i=1}^{n}X_i\\ \hat{\sigma}_n^2 : = \frac{1}{n}\sum_{i=1}^{n}(X_i-\hat{\mu}_n)^2 $$

According to the Central Limit Theorem, as $n$ becomes very large, the sample mean $\hat{\mu}_n$ will closely obey a normal distribution

$$ \hat{\mu} \sim N(\mu,\frac{\sigma^2}{n}) $$

Before we can calculate confidence intervals, the author states that since we do not know $\sigma^2$, we will make the estimation that $\sigma^2 \approx \hat{\sigma}^2$, or more precisely for an unbiased estimate $\sigma^2 \approx \frac{n}{n-1}\hat{\sigma}^2$, and we can proceed from there using standard techniques.

Now, while the author mentions the importance of $n$ sufficiently large (number of draws per simulation), there is no mention about the number of simulations and its effect on our confidence.

Is there any advantage of running $k$ simulations (performing $n$ draws each time) to obtain several sample means $\hat{\mu}_{n,1}, \hat{\mu}_{n,2}, \dots \hat{\mu}_{n,k}$, and then use the means of the means to improve our estimates and confidence regarding the unknown $\mu,\sigma$ of $X$?

Or does it suffice to just draw $n$ samples from $X$ in a single simulation, as long as $n$ is sufficiently large?

$\endgroup$
5
$\begingroup$

As long as problems regarding pseudo-random number generation are avoided (see note at the end), the two approaches ($k$ simulations with $n$ draws vs. single simulation with sufficiently large $n$) are equivalent with respect to estimating the mean. Regarding memory, observe that, in the $k$ simulations case, you need to store the sample means $\hat{\mu}_{n,1}, \dots, \hat{\mu}_{n,k}$ before performing the final mean, while this does not happen in the single simulation scenario. With modern computers, performing a single simulation with sufficiently large $n$ should not be harder than what previously described and, in fact, should save time.

The mathematical reason beyond the equivalence is linearity. To be more precise, in the $k$ simulations scenario, you calculate the "final" sample mean $\hat{\mu}$ as follows $$ \hat{\mu} = \frac{1}{k} \sum_{h=1}^k \hat{\mu}_{n,h} = \frac{1}{k} \sum_{h=1}^k \frac{1}{n} \sum_{i=1}^n X^{(h)}_i = \frac{1}{nk} \sum_{h=1}^k \sum_{i=1}^n X^{(h)}_i $$ where $X_i^{(h)}$ denotes the draw numbered $i$ at simulation $h$. This ordening is arbitrary if nothing strange happens, thus you can re-label each $X_i^{(h)}$ with a new index, say $m=1,\dots,nk$, obtaining $$ \hat{\mu} = \frac{1}{nk} \sum_{m=1}^{nk} X_m $$ But this is equivalent to performing a single simulation with $nk$ draws (obviously, the draws must be i.i.d., as already remarked).

Note: Potential problems with PRNGs are described in the Wikipedia page.

$\endgroup$
  • 1
    $\begingroup$ Great answer! I had made this realization a little after posting. And since the variance of our sampling is inversely proportional to $n$ (the number of samples), our confidence is also improved (at least, theoretically). $\endgroup$ – jII Jul 13 '14 at 8:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.