Random parameter in Monte Carlo?

Question

This is a (hopefully) pretty basic question regarding Monte Carlo Simulation. Say we have data generated by a distribution with one parameter, $\mu$. We want to estimate the expected loss of an estimation method that yields an estimate $\hat{\mu}$: $$ \mathbb{E}_\mu[L(\mu,\hat{\mu})] $$ where the subscript denotes that the expecation is taken under $\mu$ and the loss is e.g. $L(\mu,\hat{\mu}) = (\mu - \hat{\mu})^2$. In a Monte Carlo simulation, we generate many samples (say $S$), compute the estimator and the corresponding loss and then rely on a Law of Larger Numbers argument that yields $$ \frac{1}{S} \sum_{s=1}^S L(\mu,\hat{\mu}_s) \to \mathbb{E}_\mu[L(\mu,\hat{\mu})] $$ My question is: what happens when $\mu$ is itself a random draw from some population with hyperparameter $\nu$ so that $\mu_s$ differs across realizations? What happens to the loss estimate now? $$ \frac{1}{S} \sum_{s=1}^S L(\mu_s,\hat{\mu}_s) \to ? $$ Intuitively I would think that we may need to build in a second averaging step where, for each value of $\mu_s$, we average across realizations: $$ \frac{1}{S T}\sum_{s=1}^S \sum_{t=1}^T L(\mu_s,\hat{\mu}_{s,t}) $$ Would the latter converge to something like $\mathbb{E}_\nu[L(\mu,\hat{\mu})]$? And how should one in general proceed in this situation?

Answer 1

Your intuition is correct: in that case you want to compute by Monte Carlo an expectation under the joint distribution of $\mu$ and $\hat{\mu}$. Hence, if$$(\mu,\hat{\mu})\sim\nu(\mu)\times f(\hat{\mu}|\mu)$$the Monte Carlo approximation of$$\mathbb{E}_{\nu\times f}[L(\mu,\hat{\mu})]=\int L(\mu,\hat{\mu}) \nu(\mu)\times f(\hat{\mu}|\mu)\,\text{d}\mu\,\text{d}\hat{\mu}$$is obtained by simulating pairs $(\mu,\hat{\mu})$ from the joint distribution $\nu(\cdot)\times f(\cdot|\cdot)$, say $(\mu^s,\hat{\mu}^s)$, for $s=1,\ldots,S,$ and taking the average$$\frac{1}{S}\sum_{s=1}^S L(\mu^s,\hat{\mu}^s)$$ There is not particular reason for taking several realisations of $\hat{\mu}$ for a given realisation of $\mu$. (This point is discussed in more details in this other question. And on my blog.)