3
$\begingroup$

I have a conceptual confusion about the use of the expected value of a distribution.

Often, we want to estimate the most likely value of something. For example, I have X= ten observations. I know X was drawn from the poisson distribution, and I know that this poisson distribution itself has a lambda that was drawn from the lognormal distribution with mu = 4, std = .05.

So given my observations, and my knowledge about the underlying distributions, I want to estimate lambda. So Bayes theorem comes in handy, with

posterior = lambda given X

prior = lognormal distribution

likelihood = likelihood function for a poisson distribution

This can be solved (e.g., with rejection sampling) to discover that our Bayes estimator for lambda is ~4.28.

What we did, in intuitive/oversimplified terms, is take the "center of mass" of our posterior distribution, and found it was ~4.28.

My question is: if we're trying to find the most likely value of lambda, why are we taking the "center of mass" of the posterior distribution, rather than the highest point on our posterior distribution?

In this example these two are extremely similar, but you could imagine changing some assumptions in this problem, and getting a posterior distribution that is (for instance) bimodal. If this were the case, the highest point of this bimodal distribution seems like it corresponds nicely to the "most likely value of lambda," whereas the center of mass of this posterior distribution could be an extremely unlikely value of lambda (e.g., a value in the valley between the two peaks of the distribution).

Any help is appreciated!

TL;DR: When estimating the 'most likely value' for a parameter, why do we ever want to do so by finding the 'center of mass' (expected value) from that parameter's distribution? Shouldn't we always want the 'highest point' (MLE) on that distribution?

Improve this question
$\endgroup$
2
  • $\begingroup$ @whuber Yep - I re-read the question after posting and realized that I was responding to an implicit comparison of MLE vs posterior inference rather than his substantive concern. $\endgroup$ – Sycorax♦ May 2 '14 at 17:35
  • $\begingroup$ Never heard the term "center of mass" before. Is that the same thing as mean and expected value? $\endgroup$ – confused Nov 19 '19 at 17:29
1
$\begingroup$

Sometimes we also us the median of the posterior distribution which gives 3 commonly used single number summaries of the posterior (mean/expected value, median, and mode/highest point). All three have advantages and disadvantages. We can also use the entire posterior (best option when reasonable) or a credible interval based on the posterior (there can be infinitely many of these, but the equal tail and highest posterior density intervals are probably the most common).

One place where we would not want to use the mode/highest point is if the posterior is an exponential distribution, or a beta distribution with parameters less than 1.

Probably the most common reason to use the expected value/mean or the median instead of the highest point/mode is convenience of calculation. For simple problems with a conjugate prior there may not be much difference is the effort to calculate the different measures, but as problems become more complicated we will often not have an exact idea of the posterior, but will rather use some form of Monte Carlo method to sample from the posterior. In that case the best estimate for the mean of the posterior is the mean of the samples, nice and simple, but the highest point/mode is more difficult (and if we just look at the mode of the samples then there might not be one, or what we see may be influenced more by a quirk of rounding than the true properties of the posterior.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the response! I'm with you until that last paragraph. My understanding of these sorts of problems is that it's relatively easy to find the pdf (let's call it f(x)) for your posterior distribution. What's hard is integrating over f(x), since analytic solutions are rarely available. So we have to resort to computationally expensive methods to sample from f(x). Meanwhile, finding the max of f(x) seems as straightforward as plugging some x values in a plausible range into f(x) and using a built-in max() function. I must be missing some pretty basic assumptions here, sorry. $\endgroup$ – user44845 May 3 '14 at 20:59
  • $\begingroup$ @jwdink, for simple cases your are correct, and possibly even some medium cases, but some of the harder cases (including many real problems) it is not that simple. In some cases you can easily define the correct conditional distributions to do gibbs sampling, but difficult to find the actual posterior. In other cases you may find the posterior, but it has multiple local modes over many dimensions, you may find a point that has a gradient of 0, but how do you know that it is the maximum rather than a local high or saddle point? $\endgroup$ – Greg Snow May 6 '14 at 19:03
2
$\begingroup$

You are necessarily throwing away information when you convert a posterior distribution to a point estimate. The specific choice of point estimate (expected value or maximum aposteriori (MAP) estimate) depends on your purpose. What are you using the resulting point estimate for? I think the best case for using the expected value is the useful property of linearity. That is,

$E\left(y_1 + y_2 + ... + y_n\right) = E\left(y_1\right) + E\left(y_2\right) + ... + E\left(y_n\right)$

This property is useful if you are trying to make predictions about a population. For example, if you are modeling cost for an insured population, you can get the expected cost for the population by adding together the expected costs of the individual members of that population.

Improve this answer
$\endgroup$
1
$\begingroup$

A Bayesian point estimator is a summary of the posterior used for a specific purposed and justified by an optimisation principle associated with a utility function $U$. The point estimate $\hat\theta(\cdot)$ is the function that maximises the posterior expected utility $$\mathbb E[U(\hat\theta,\theta)|x]=\int U(\hat\theta,\theta)\,\pi(\theta|x)\,\text d\theta$$ The posterior mean and the posterior median are associated with L² and L¹ utilities for instance. The posterior mode is not clearly associated with a utility function and its qualification as a Bayesian estimator is disputed.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy