# Sampling from an approximate distribution to estimate posterior mean

Suppose I have a model with parameters $$\theta = (\theta_1,\theta_2)$$ and observed data $$x=(x_1,x_2,\ldots,x_k)$$. I want to estimate the posterior mean $$\hat{\theta}_1 = \mathbb{E}[\theta_1|x]$$.

The issue is that I can't sample from $$p(\theta_1|x)$$ to estimate $$\hat{\theta}_1$$ by the sample mean because I do not count on an expression for the likelihood $$p(x|\theta_1)$$. I do, however, count on an expression for $$p(x|\theta_1,\theta_2) = p(x|\theta)$$ as well as on a prior $$\pi(\theta_1)$$. I can also find the maximum likelihood estimator for $$\theta_2$$ (let's call it $$\hat{\theta}_2$$).

Let $$\hat{p}(x|\theta_1)=p\left(x|\theta_1,\hat{\theta}_2\right)$$. Then, I could sample from an approximate posterior distribution

$$\hat{p}(\theta_1|x) = \frac{1}{C}\frac{\hat{p}(x|\theta_1)\pi(\theta_1)}{p(x)}$$

where $$C$$ is some normalization constant.

If I compute the mean of $$N$$ samples from $$\hat{p}(\theta_1|x)$$, would that be a "reasonable estimator" (in any sense) for $$\mathbb{E}[\theta_1|x]$$? Is there any literature I could resort to that does something similar to this?

• Don't mix frequentist and Bayesian approaches. Go fully Bayesian (identify a joint prior) Commented Apr 30, 2023 at 23:02
• @Glen_b Now that you mention it, I see that this approach may be quite weird. But I can't see why it shouldn't (or why it should) work... I was wondering whether any analysis could be done to study the estimator I proposed, or if there are any resources I could go check out that is related to this. Commented Apr 30, 2023 at 23:28
• Analysis could be done to look at frequentist properties, sure. In suitable circumstances/with suitable assumptions you may be also able to look at some Bayesian properties. e.g. if you treat the joint prior as independent, put a flat prior on the second parameter and hence look at the MLE as a MAP estimate, you may be able to make some progress there (I haven't really thought about what issues you'd run into but if there are some they might be solvable; you might at least be able to talk about the problems). ... ctd Commented May 1, 2023 at 0:31
• ctd... I don't know of any resources to point to; if I was somehow convinced to make such an attempt I'd be doing it from scratch. Commented May 1, 2023 at 0:32

The issue with the question is that the expression$$\mathbb E[\theta_1|x]$$is not well-defined:
1. either $$(\theta_1,\theta_2)$$ is considered a random vector with joint prior $$\pi(\theta_1,\theta_2)$$, in which case $$\mathbb E[\theta_1|x]=\int \theta_1 \pi(\theta_1,\theta_2|x)\,\text d(\theta_1,\theta_2)$$only depends on $$x$$ and can be computed
2. or $$\theta_1$$ is considered a random variable with prior $$\pi(\theta_1)$$, while $$\theta_2$$ is an unknown fixed value, in which case $$\mathbb E_{\theta_2}[\theta_1|x]=\int \theta_1 \pi_{\theta_2}(\theta_1|x)\,\text d\theta_1$$ is indexed by $$\theta_1$$, hence cannot be computed when $$\theta_2$$ is unknown.
• Thanks for your answer! It's really clear now. My OP meant to tackle the second case you mention. Namely, given that $\theta_2$ is unknown but it does have a value, I thought of plugging in its MLE instead. I have edited the question, does it make sense now? Would my OP be answereable under this setting? Commented May 1, 2023 at 15:30
• What is the meaning of "reasonable" in that context mixing a Bayes estimate for $\theta_1$ and the MLE for $\theta_2$? Consistency as an estimator of $\theta_1$? Commented May 1, 2023 at 15:43
• I know it's vague, but I was looking for any result regarding desirable properties of an estimator (consistency, unbias, efficiency...). But now that I think about it (after reading your answer), would it make sense to analyze these properties for an estimator of $\theta_1$, given that it is a random variable (under this Bayesian framework)? Commented May 1, 2023 at 16:31
• Maybe it would make sense to study properties of an estimator of $\hat{\theta}_1 = \mathbb E[\theta_1|x]$, which is an unknown fixed value. The estimator would be the empirical mean of $N$ samples drawn from the $\hat{p}(\theta_1|x)$ I defined in the OP. Would that make more sense? Commented May 1, 2023 at 16:33