0
$\begingroup$

Let us say we have:

  1. Data $X$
  2. Parameter that we are trying to estimate is $\Theta$

The Bayesian estimation method is to

  1. Assume a prior on $\Theta$
  2. Sample $x$ from $X$
  3. Use Bayes theorem. Compute the posterior $\Theta\mid(X=x)$

Now, in case a point estimate is to be reported then one way is to report lowest mean square estimate $\hat{\Theta} = E[\Theta \mid X]$. This is supposed to be the parameter that has lowest mean squared error.

The error being defined as $\bar{\Theta}=\hat{\Theta} - \Theta$.

Now, two supposedly intuitive properties of $\bar{\Theta}$ are

  1. $E[\bar{\Theta}] = 0$
  2. $E[\bar{\Theta} \mid X] = 0$

Whenever I think of expected values I think there is an inherent distribution that the expectation is computed using. What is the distribution over which $E[\bar{\Theta}] = 0$ is computed? There are two sources of randomness namely $X$ and $\Theta$.

Moreover, Expectations should be able to be estimated by iterated sampling. That procedure, I think, would like the following in the case of $E[\bar{\Theta} \mid X]$

  1. Sample $x$ from $X$.
  2. Compute posterior $\Theta \mid (X = x)$
  3. Compute $\hat{\theta} = E[\Theta \mid X = x]$. The expectation of the posterior computed above
  4. Sample $\theta$ from posterior $\Theta \mid (X = x)$
  5. Compute $\bar{\theta}=\hat{\theta} - \theta$

If you iterate over these 5 steps large number of times then the average of $\bar{\theta}$ would converge to 0

What would that procedure look like for $E[\bar{\Theta}]$? My attempt:

  1. Sample $x$ from $X$
  2. Sample $\theta$ from Prior $\Theta$
  3. Compute posterior $\Theta \mid (X = x)$
  4. Compute $\hat{\theta} = E[\Theta \mid X = x]$. The expectation of the posterior computed above
  5. Compute $\bar{\theta}=\hat{\theta} - \theta$

If you iterate over these 5 steps large number of times then the average of $\bar{\theta}$ would converge to 0

Am I right?

$\endgroup$
4
  • $\begingroup$ This confuses me: "Now, in case a point estimate is to be reported then one way is to report lowest mean square estimate $\hat{\Theta} = E[\Theta \mid X]$. This is supposed to be the parameter that has lowest mean squared error." To what degree is that a statement and to what degree a question? You seem to say that you want to report the posterior mean and make it the subject of the analysis thereafter. Or do you rather want a $X$-MSE minimizer? Or a $\Theta$-MSE minimizer? These three are not always the same. $\endgroup$ Jun 30 at 21:25
  • $\begingroup$ The $X$-MSE minimizer is sometimes the same as the maximum likelihood estimator $\hat{\Theta}_\mathrm{ML}$ and the posterior mode is the maximum a posteriori estimator $\hat{\Theta}_\mathrm{MAP}$. $\endgroup$ Jun 30 at 21:32
  • $\begingroup$ MAP makes sense for discrete distributions imo. You want to minimize the probability of being wrong. I don't want to maximize the likelihood of data. That is not Bayesian. I am looking for an estimator that minimises the expected error over $\Theta$ $\endgroup$ Jun 30 at 21:35
  • $\begingroup$ so when writing MSE, you had $(E[(\hat{Θ}−Θ)^\intercal(\hat{Θ}−Θ)|X]$ in mind? The not so nice things about this are: 1. some dimensions in Θ can be more or less meaningless than others, Distance in parameter space is therefore meaningless. 2. your computations so far only work for the posterior mean. I suggest that we forget about all things MSE, and that you indicate this up in your post. $\endgroup$ Jun 30 at 22:17

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.