1
$\begingroup$

This is a (hopefully) pretty basic question regarding Monte Carlo Simulation. Say we have data generated by a distribution with one parameter, $\mu$. We want to estimate the expected loss of an estimation method that yields an estimate $\hat{\mu}$: $$ \mathbb{E}_\mu[L(\mu,\hat{\mu})] $$ where the subscript denotes that the expecation is taken under $\mu$ and the loss is e.g. $L(\mu,\hat{\mu}) = (\mu - \hat{\mu})^2$. In a Monte Carlo simulation, we generate many samples (say $S$), compute the estimator and the corresponding loss and then rely on a Law of Larger Numbers argument that yields $$ \frac{1}{S} \sum_{s=1}^S L(\mu,\hat{\mu}_s) \to \mathbb{E}_\mu[L(\mu,\hat{\mu})] $$ My question is: what happens when $\mu$ is itself a random draw from some population with hyperparameter $\nu$ so that $\mu_s$ differs across realizations? What happens to the loss estimate now? $$ \frac{1}{S} \sum_{s=1}^S L(\mu_s,\hat{\mu}_s) \to ? $$ Intuitively I would think that we may need to build in a second averaging step where, for each value of $\mu_s$, we average across realizations: $$ \frac{1}{S T}\sum_{s=1}^S \sum_{t=1}^T L(\mu_s,\hat{\mu}_{s,t}) $$ Would the latter converge to something like $\mathbb{E}_\nu[L(\mu,\hat{\mu})]$? And how should one in general proceed in this situation?

$\endgroup$
1
$\begingroup$

Your intuition is correct: in that case you want to compute by Monte Carlo an expectation under the joint distribution of $\mu$ and $\hat{\mu}$. Hence, if$$(\mu,\hat{\mu})\sim\nu(\mu)\times f(\hat{\mu}|\mu)$$the Monte Carlo approximation of$$\mathbb{E}_{\nu\times f}[L(\mu,\hat{\mu})]=\int L(\mu,\hat{\mu}) \nu(\mu)\times f(\hat{\mu}|\mu)\,\text{d}\mu\,\text{d}\hat{\mu}$$is obtained by simulating pairs $(\mu,\hat{\mu})$ from the joint distribution $\nu(\cdot)\times f(\cdot|\cdot)$, say $(\mu^s,\hat{\mu}^s)$, for $s=1,\ldots,S,$ and taking the average$$\frac{1}{S}\sum_{s=1}^S L(\mu^s,\hat{\mu}^s)$$ There is not particular reason for taking several realisations of $\hat{\mu}$ for a given realisation of $\mu$. (This point is discussed in more details in this other question. And on my blog.)

$\endgroup$
1
  • $\begingroup$ That was quick - thank you so much, also for the link! $\endgroup$ Oct 31 '16 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.