1
$\begingroup$

Suppose I have a model with parameters $\theta = (\theta_1,\theta_2)$ and observed data $x=(x_1,x_2,\ldots,x_k)$. I want to estimate the posterior mean $\hat{\theta}_1 = \mathbb{E}[\theta_1|x]$.

The issue is that I can't sample from $p(\theta_1|x)$ to estimate $\hat{\theta}_1$ by the sample mean because I do not count on an expression for the likelihood $p(x|\theta_1)$. I do, however, count on an expression for $p(x|\theta_1,\theta_2) = p(x|\theta)$ as well as on a prior $\pi(\theta_1)$. I can also find the maximum likelihood estimator for $\theta_2$ (let's call it $\hat{\theta}_2$).

Let $\hat{p}(x|\theta_1)=p\left(x|\theta_1,\hat{\theta}_2\right)$. Then, I could sample from an approximate posterior distribution

$$\hat{p}(\theta_1|x) = \frac{1}{C}\frac{\hat{p}(x|\theta_1)\pi(\theta_1)}{p(x)}$$

where $C$ is some normalization constant.

If I compute the mean of $N$ samples from $\hat{p}(\theta_1|x)$, would that be a "reasonable estimator" (in any sense) for $\mathbb{E}[\theta_1|x]$? Is there any literature I could resort to that does something similar to this?

$\endgroup$
4
  • 1
    $\begingroup$ Don't mix frequentist and Bayesian approaches. Go fully Bayesian (identify a joint prior) $\endgroup$
    – Glen_b
    Apr 30 at 23:02
  • $\begingroup$ @Glen_b Now that you mention it, I see that this approach may be quite weird. But I can't see why it shouldn't (or why it should) work... I was wondering whether any analysis could be done to study the estimator I proposed, or if there are any resources I could go check out that is related to this. $\endgroup$
    – Tendero
    Apr 30 at 23:28
  • 1
    $\begingroup$ Analysis could be done to look at frequentist properties, sure. In suitable circumstances/with suitable assumptions you may be also able to look at some Bayesian properties. e.g. if you treat the joint prior as independent, put a flat prior on the second parameter and hence look at the MLE as a MAP estimate, you may be able to make some progress there (I haven't really thought about what issues you'd run into but if there are some they might be solvable; you might at least be able to talk about the problems). ... ctd $\endgroup$
    – Glen_b
    May 1 at 0:31
  • $\begingroup$ ctd... I don't know of any resources to point to; if I was somehow convinced to make such an attempt I'd be doing it from scratch. $\endgroup$
    – Glen_b
    May 1 at 0:32

1 Answer 1

3
$\begingroup$

The issue with the question is that the expression$$\mathbb E[\theta_1|x]$$is not well-defined:

  1. either $(\theta_1,\theta_2)$ is considered a random vector with joint prior $\pi(\theta_1,\theta_2)$, in which case $$\mathbb E[\theta_1|x]=\int \theta_1 \pi(\theta_1,\theta_2|x)\,\text d(\theta_1,\theta_2)$$only depends on $x$ and can be computed

  2. or $\theta_1$ is considered a random variable with prior $\pi(\theta_1)$, while $\theta_2$ is an unknown fixed value, in which case $$\mathbb E_{\theta_2}[\theta_1|x]=\int \theta_1 \pi_{\theta_2}(\theta_1|x)\,\text d\theta_1$$ is indexed by $\theta_1$, hence cannot be computed when $\theta_2$ is unknown.

$\endgroup$
4
  • $\begingroup$ Thanks for your answer! It's really clear now. My OP meant to tackle the second case you mention. Namely, given that $\theta_2$ is unknown but it does have a value, I thought of plugging in its MLE instead. I have edited the question, does it make sense now? Would my OP be answereable under this setting? $\endgroup$
    – Tendero
    May 1 at 15:30
  • 1
    $\begingroup$ What is the meaning of "reasonable" in that context mixing a Bayes estimate for $\theta_1$ and the MLE for $\theta_2$? Consistency as an estimator of $\theta_1$? $\endgroup$
    – Xi'an
    May 1 at 15:43
  • $\begingroup$ I know it's vague, but I was looking for any result regarding desirable properties of an estimator (consistency, unbias, efficiency...). But now that I think about it (after reading your answer), would it make sense to analyze these properties for an estimator of $\theta_1$, given that it is a random variable (under this Bayesian framework)? $\endgroup$
    – Tendero
    May 1 at 16:31
  • $\begingroup$ Maybe it would make sense to study properties of an estimator of $\hat{\theta}_1 = \mathbb E[\theta_1|x]$, which is an unknown fixed value. The estimator would be the empirical mean of $N$ samples drawn from the $\hat{p}(\theta_1|x)$ I defined in the OP. Would that make more sense? $\endgroup$
    – Tendero
    May 1 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.