Advantage of multiple simulations in old-fashioned Monte Carlo?

Question

The spirit of this question comes from "Ordinary Monte Carlo", also known as "good old-fashioned Monte Carlo"

Suppose I have a random variable $X$, with
$$\mu := E[X]\\ \sigma^2:=Var[X] $$

Both are unknown values, because the probability distribution function of $X$ is unknown (or the computations are intractable).

Either way, suppose we can somehow simulate $n$ draws $X_1,X_2,\dots,X_n$ (these are independent and identically distributed) from the distribution of $X$. Let us define the sample parameters

$$ \hat{\mu}_n := \frac{1}{n}\sum_{i=1}^{n}X_i\\ \hat{\sigma}_n^2 : = \frac{1}{n}\sum_{i=1}^{n}(X_i-\hat{\mu}_n)^2 $$

According to the Central Limit Theorem, as $n$ becomes very large, the sample mean $\hat{\mu}_n$ will closely obey a normal distribution

$$ \hat{\mu} \sim N(\mu,\frac{\sigma^2}{n}) $$

Before we can calculate confidence intervals, the author states that since we do not know $\sigma^2$, we will make the estimation that $\sigma^2 \approx \hat{\sigma}^2$, or more precisely for an unbiased estimate $\sigma^2 \approx \frac{n}{n-1}\hat{\sigma}^2$, and we can proceed from there using standard techniques.

Now, while the author mentions the importance of $n$ sufficiently large (number of draws per simulation), there is no mention about the number of simulations and its effect on our confidence.

Is there any advantage of running $k$ simulations (performing $n$ draws each time) to obtain several sample means $\hat{\mu}_{n,1}, \hat{\mu}_{n,2}, \dots \hat{\mu}_{n,k}$, and then use the means of the means to improve our estimates and confidence regarding the unknown $\mu,\sigma$ of $X$?

Or does it suffice to just draw $n$ samples from $X$ in a single simulation, as long as $n$ is sufficiently large?

Answer 1

As long as problems regarding pseudo-random number generation are avoided (see note at the end), the two approaches ($k$ simulations with $n$ draws vs. single simulation with sufficiently large $n$) are equivalent with respect to estimating the mean. Regarding memory, observe that, in the $k$ simulations case, you need to store the sample means $\hat{\mu}_{n,1}, \dots, \hat{\mu}_{n,k}$ before performing the final mean, while this does not happen in the single simulation scenario. With modern computers, performing a single simulation with sufficiently large $n$ should not be harder than what previously described and, in fact, should save time.

The mathematical reason beyond the equivalence is linearity. To be more precise, in the $k$ simulations scenario, you calculate the "final" sample mean $\hat{\mu}$ as follows $$ \hat{\mu} = \frac{1}{k} \sum_{h=1}^k \hat{\mu}_{n,h} = \frac{1}{k} \sum_{h=1}^k \frac{1}{n} \sum_{i=1}^n X^{(h)}_i = \frac{1}{nk} \sum_{h=1}^k \sum_{i=1}^n X^{(h)}_i $$ where $X_i^{(h)}$ denotes the draw numbered $i$ at simulation $h$. This ordening is arbitrary if nothing strange happens, thus you can re-label each $X_i^{(h)}$ with a new index, say $m=1,\dots,nk$, obtaining $$ \hat{\mu} = \frac{1}{nk} \sum_{m=1}^{nk} X_m $$ But this is equivalent to performing a single simulation with $nk$ draws (obviously, the draws must be i.i.d., as already remarked).

Note: Potential problems with PRNGs are described in the Wikipedia page.