the simplest Bayesian estimate?

Question

The textbook A first Course in Machine Learning, by Rogers & Girolami first covers ML, $$ \arg \max_M P(D|M) $$ then MAP, $$ \arg\max_M p(D|M) P(M) $$

However in transition to the Bayesian case, the example it gives is of estimating data^1, rather than estimate a model parameter, so the example is not parallel to the previous two cases.

My question: Is this a simple valid example of a Bayesian parameter estimate: $$ M_{\text{Bayesian}} = \int \frac{P(D|M) P(M)}{P(D)} dM $$ It looks wrong to me, in that it is somewhat saying that "the parameter M is an average of the parameter M".

End of question.

Footnote 1: Specifically the introduction Bayesian example is $$ p(x|D) = \int p(x|M) p(M|D) dM $$

Answer 1

First, the notation of $M$ for the parameter of the model is quite confusing, as it seems like a model index. If I denote the parameter by $\theta$, the density of the observation $x$ by $p(x|\theta)$, and the prior probability by $\pi(\theta)$, then $$\int \frac{P(D|M) P(M)}{P(D)} dM$$ translates into $$\int \dfrac{p(x|\theta)\pi(\theta)}{\int p(x|\theta)\pi(\theta)\text{d}\theta}\text{d}\theta=\dfrac{\int p(x|\theta)\pi(\theta)\text{d}\theta}{\int p(x|\theta)\pi(\theta)\text{d}\theta}=1$$ hence this is obviously not correct.

When computing a posterior expectation of the parameter $\theta$ as a possible Bayesian estimator (associated with a quadratic loss function) $$\hat{\theta}(x)=\int \theta\,\dfrac{p(x|\theta)\pi(\theta)}{\int p(x|\theta)\pi(\theta)\text{d}\theta}\text{d}\theta$$ the estimator (lhs) is a function of $x$, equal to an average over the parameter space (rhs) so there is no contradiction of the type "the parameter M is an average of the parameter M". The rhs is not the parameter behind the data (or the parameter generating the data) but an estimate of the parameter: while they live in the same space, they have different interpretations.

Answer 2

No, this is not an example of a Bayesian parameter estimate. The notation is a bit wonky compared to more standard notation.

To clean it up a bit and put it into a more standard notation using $\theta\in\Theta$ for the potential parameters, $x\in\chi$ for the sample in the sample space and $\tilde{x}\in\tilde{\chi}$ for the potential future sample in the potential future sample space we get the map estimator as $$\sup_{\theta}p(x|\theta)p(\theta).$$

Please note that models are considered parameters in Bayesian thinking. The posterior distribution is defined as $$p(\theta|x)=\frac{p(x|\theta)p(\theta)}{\int_{\theta\in\Theta}p(x|\theta)p(\theta)\mathrm{d}\theta},\forall\theta\in\Theta.$$ This is your parameter estimator and it is a density. The predictive distribution, though is $$p(\tilde{x}|x)=\int_{\theta\in\Theta}p(\tilde{x}|\theta)p(\theta|x)\mathrm{d}\theta,\forall\tilde{x}\in\tilde{\chi}.$$

Note that $\theta$ has vanished on the left-hand side. It is no longer part of the discussion. It doesn't matter what the true value is; it only matters what the sample was. You have moved from working in the parameter space as random into the future sample space as random. $\theta\in\Theta$ is about uncertainty as to the true value of the parameter, but $\tilde{x}\in\tilde{\chi}$ is about chance drawings of future values of $\tilde{x}$ as there is no uncertainty remaining regarding $\theta\in\Theta$.

Now let us note that the MAP estimator minimizes the all-or-nothing cost function. You can do this because there is uncertainty about $\theta\in\Theta$, but you can also do that for $\tilde{x}\in\tilde{\chi}$ as you do not know the outcome of chance, but with a different meaning. So choosing $\tilde{x}$ as the supremum of the set of possible solutions as the most likely draw gives you the mode of the prediction. Of course, there are other possible point estimators such as the mean of the prediction and other cost functions.

This can be a very valuable tool when you are operating on variables outside the exponential family of distributions because they will have no sufficient statistic. If you use a Frequentist solution on distributions that lack a sufficient statistic, then you must lose information in your parameter estimate. However, because the Bayesian likelihood function is always minimally sufficient, you end up using all of the available information for a prediction.

The entire posterior density is sufficient even though no point, including the MAP, is sufficient. By integrating the uncertainty out and imposing a cost function onto the prediction, you have captured the available information about future possible values of $\tilde{x}$ without an unnecessary information loss and have reduced it to a single point. This is valuable if the solution is that you need 13 pounds of flour instead of a density of possible future amounts of needed flour.

One last note about the posterior mean, $\hat{\theta}_{mean}(x)$ is the average of your beliefs about $\theta$ and is not an average of $\theta$. It is your average of $\theta.$ I may have a different one. This is a subjective system of statistics and not an objective one like the Frequentist method. If we both see the same data, we will get different summary statistics if we have different prior densities.

EDIT Although it is oversimplified, a statistic is sufficient if you could throw away the raw data and not lose any information. For the normal distribution, this allows you to store the parameter estimates and ignore what could be billions of pieces of data.

A statistic is admissible if there is no less risky manner from which to calculate an estimate. For example, both the median and the mean are estimators for the center of location of the univariate normal distribution. However, we almost always use the mean. This is because there is more risk of using the median and being wrong than the mean and being wrong. Estimates always have a cost of being wrong if you use them in real-world decisions. Preferably, you would always choose the least risky way of finding them.

Notice that you are not integrating over $x$ or $\tilde{x}$, but rather over $\theta\in\Theta$ in all of these calculations. You can usually use $\hat{\theta}$ for $\hat{x}$, but there are cases where you would lose information in doing so and add risk at the same time.

I am assuming your original training was in Frequentist methods or using the method of maximum likelihood from your comments. Bayesian estimates are entire distributions and not generally points. Point estimators are important in the other two primary fields of statistics because you can form a hypothesis test using them, whereas you cannot use a Bayesian point estimate to test a hypothesis.

In Frequentist methods $\bar{x}=5$ is important in many different ways. In Bayesian statistics, $\hat{\theta}(x)=5$ is just raw information. It doesn't really mean anything unless you have a reason that $\hat{\theta}(x)$ is explicitly useful.

For example, let us imagine you are searching for a submarine one location at a time. Then you would want to use $\hat{\theta}_{MAP}(x)$ because it is the single most likely answer. On the other hand, if you were gambling on the batting average of Corey Dickerson, then you would want to use $\hat{\theta}_{Mean}(x)$. This would preserve the good mathematical properties from minimizing quadratic loss.

Nonetheless, while you could minimize the mean of the posterior for the gamble, wouldn't you be better off with finding the mean of the prediction of the future? They are not always the same thing.

Consider the case where your data is multimodal with one value just slightly denser than another. Let us imagine that the two values are one and seven and that there is no data at all between three and five. You choose one because it is the most likely answer and ignore seven, yet it is almost the same density.

As a rule of thumb, Frequentist point estimators are going to be unimodal, this will not be true for Bayesian methods. Consider the case where you are estimating customer behavior and gambling inventory levels on it. You have decided to ignore the posterior density and only look for one point. It turns out there are two modes because customer behavior on rainy days and sunny days are different. You did not collect weather data in your set and so now you have two modes. More often, in your sample, it rains rather than not. You buy one unit of inventory as it is the MAP estimator, but you have a long string of sunny days and a completely empty store with nothing to sell.

If you use Bayesian methods you are not looking for points, you are looking at entire distributions. You use points either to describe the data or to make use in a decision. Bayesian methods are used less because they are very inconvenient.